[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:36 dist-test-slave-f9rl krb5kdc[19650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:36 dist-test-slave-f9rl krb5kdc[19650](info): set up 2 sockets
Sep 25 10:02:36 dist-test-slave-f9rl krb5kdc[19650](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:02:37.379415 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:37.517952 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36899
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:36899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:37.603026 19663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:37.603291 19663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:37.603329 19663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:37.604543 19663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:37.604614 19663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:37.604638 19663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:37.604660 19663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:37.604691 19663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:37.606314 19663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:36899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:37.606637 19663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:37.606907 19663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:02:37.612790 19663 server_base.cc:1034] running on GCE node
W20260925 10:02:37.612707 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.612758 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.612732 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:37.613229 19663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:37.613682 19663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:37.614944 19663 hybrid_clock.cc:648] HybridClock initialized: now 1790330557614934 us; error 52 us; skew 500 ppm
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:37.618062 19663 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:37.618681 19663 webserver.cc:460] Webserver started at http://127.19.46.254:40707/ using document root <none> and password file <none>
I20260925 10:02:37.619036 19663 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:37.619122 19663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:37.619272 19663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:37.620461 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "b2d14a37fae843b0b686cd2b667b92ac"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "48b2ac2b4fdc9192aea26c1396d47a5a"
server_key_iv: "183f8dad371978a7d23bb26f040a18be"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.620863 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "b2d14a37fae843b0b686cd2b667b92ac"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "48b2ac2b4fdc9192aea26c1396d47a5a"
server_key_iv: "183f8dad371978a7d23bb26f040a18be"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.622424 19663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:02:37.623337 19678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:37.623593 19663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:37.623714 19663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "b2d14a37fae843b0b686cd2b667b92ac"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "48b2ac2b4fdc9192aea26c1396d47a5a"
server_key_iv: "183f8dad371978a7d23bb26f040a18be"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.623797 19663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:37.642921 19663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:37.645284 19663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:37.645435 19663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:37.649241 19663 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:36899
I20260925 10:02:37.649314 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:36899 every 8 connection(s)
I20260925 10:02:37.649775 19663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:37.650789 19731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:37.653216 19731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac: Bootstrap starting.
I20260925 10:02:37.653873 19731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:37.654191 19731 log.cc:824] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:37.654274 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 19663
I20260925 10:02:37.654394 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:37.655071 19731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac: No bootstrap required, opened a new log
I20260925 10:02:37.655108 19643 external_mini_cluster.cc:1428] Setting key 6298860165f6bbb884884639bcfe5070
I20260925 10:02:37.657140 19731 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } }
I20260925 10:02:37.657315 19731 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:37.657357 19731 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d14a37fae843b0b686cd2b667b92ac, State: Initialized, Role: FOLLOWER
I20260925 10:02:37.657541 19731 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } }
I20260925 10:02:37.657634 19731 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:37.657676 19731 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:37.657732 19731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:37.658655 19731 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } }
I20260925 10:02:37.658788 19731 leader_election.cc:302] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d14a37fae843b0b686cd2b667b92ac; no voters:
I20260925 10:02:37.658972 19731 leader_election.cc:288] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:37.659003 19736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:37.659106 19736 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [term 1 LEADER]: Becoming Leader. State: Replica: b2d14a37fae843b0b686cd2b667b92ac, State: Running, Role: LEADER
I20260925 10:02:37.659212 19736 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } }
I20260925 10:02:37.659257 19731 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:37.659574 19736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d14a37fae843b0b686cd2b667b92ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } } }
I20260925 10:02:37.659745 19736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:37.659940 19736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d14a37fae843b0b686cd2b667b92ac. Latest consensus state: current_term: 1 leader_uuid: "b2d14a37fae843b0b686cd2b667b92ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d14a37fae843b0b686cd2b667b92ac" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36899 } } }
I20260925 10:02:37.660058 19736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:37.660429 19744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:37.660985 19744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:37.662554 19744 catalog_manager.cc:1347] Generated new cluster ID: a8ca1a7a662a4aaea5afb8a025c9fc7a
I20260925 10:02:37.662608 19744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:37.674078 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:37.656056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.19.46.254:36899)
0925 10:02:37.656252 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:37.656260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:37.656543 (+ 283us) server_negotiation.cc:407] Connection header received
0925 10:02:37.658515 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:37.658520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:37.658563 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:37.658715 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:37.661027 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.662514 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.664053 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.664694 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.664815 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:37.669634 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:37.669640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:37.669641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:37.669662 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:37.671801 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:37.672321 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:37.672324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:37.672325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:37.672489 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:37.672812 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:37.672813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:37.672813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:37.673164 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:37.673335 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:37.673660 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 10:02:37.673819 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260925 10:02:37.677199 19744 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:37.677851 19744 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:37.680899 19744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d14a37fae843b0b686cd2b667b92ac: Generated new TSK 0
I20260925 10:02:37.681114 19744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:37.713572 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:37.798439 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:37.798655 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:37.798681 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:37.799954 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:37.800006 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:37.800034 19757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:37.801714 19757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:37.801978 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:37.802260 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:37.807754 19763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.807754 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.807839 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:37.808132 19757 server_base.cc:1034] running on GCE node
I20260925 10:02:37.808375 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:37.808666 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:37.809847 19757 hybrid_clock.cc:648] HybridClock initialized: now 1790330557809840 us; error 73 us; skew 500 ppm
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:37.812930 19757 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:37.813771 19757 webserver.cc:460] Webserver started at http://127.19.46.193:43841/ using document root <none> and password file <none>
I20260925 10:02:37.814077 19757 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:37.814162 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:37.814488 19757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:37.815831 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "8f72346eeaff411db5b2d5f345f6b9e8"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "810fa95f4ccca3a23e97fcc6425404d9"
server_key_iv: "53b3fdcce97a89e8d9ab76e15569979c"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.816268 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "8f72346eeaff411db5b2d5f345f6b9e8"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "810fa95f4ccca3a23e97fcc6425404d9"
server_key_iv: "53b3fdcce97a89e8d9ab76e15569979c"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.817904 19757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 10:02:37.819061 19772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:37.819238 19757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:37.819337 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "8f72346eeaff411db5b2d5f345f6b9e8"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "810fa95f4ccca3a23e97fcc6425404d9"
server_key_iv: "53b3fdcce97a89e8d9ab76e15569979c"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.819455 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:37.828265 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:37.830819 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:37.830948 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:37.831209 19757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:37.831552 19757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:37.831586 19757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:37.831610 19757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:37.831622 19757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:37.837760 19757 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:40667
I20260925 10:02:37.837842 19885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:40667 every 8 connection(s)
I20260925 10:02:37.838198 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:37.838783 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 19757
I20260925 10:02:37.838888 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:37.839062 19643 external_mini_cluster.cc:1428] Setting key ab25837566e6898814bdd6ec687e2ef3
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:02:37.848857 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:37.838992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:55615 (local address 127.19.46.254:36899)
0925 10:02:37.839065 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:37.839067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:37.839290 (+ 223us) server_negotiation.cc:407] Connection header received
0925 10:02:37.839914 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:37.839915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:37.839929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:37.839974 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:37.841114 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.841736 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.842752 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.842977 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.843066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:37.845669 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:37.845670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:37.845671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:37.845680 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:37.847258 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:37.847716 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:37.847717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:37.847718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:37.847793 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:37.848132 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:37.848133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:37.848133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:37.848272 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:37.848347 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:37.848657 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 10:02:37.848757 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:02:37.849061 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:37.838983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36899 (local address 127.19.46.193:55615)
0925 10:02:37.839156 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:37.839201 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:37.839806 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:37.840048 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:37.840052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:37.840795 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:37.841002 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:37.841007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.841834 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.841835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:37.842657 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:37.842658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:37.843079 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:37.843080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:37.843214 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:37.843771 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:37.843779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:37.845545 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:37.847355 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:37.847367 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:37.847370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:37.847630 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:37.847928 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:37.847929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:37.847930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:37.848056 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:37.848348 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:37.848357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:37.848510 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:02:37.848619 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:02:37.848809 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260925 10:02:37.849653 19886 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36899
I20260925 10:02:37.849756 19886 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:37.849915 19886 heartbeater.cc:511] Master 127.19.46.254:36899 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
I20260925 10:02:37.850960 19695 ts_manager.cc:194] Registered new tserver with Master: 8f72346eeaff411db5b2d5f345f6b9e8 (127.19.46.193:40667)
I20260925 10:02:37.851702 19695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:55615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:37.869292 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:37.954087 19891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:37.954314 19891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:37.954340 19891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:37.955493 19891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:37.955538 19891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:37.955567 19891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:37.956974 19891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:37.957165 19891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:37.957350 19891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:37.963124 19897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.963153 19899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:37.963153 19896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:37.963205 19891 server_base.cc:1034] running on GCE node
I20260925 10:02:37.963615 19891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:37.963899 19891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:37.965065 19891 hybrid_clock.cc:648] HybridClock initialized: now 1790330557965047 us; error 30 us; skew 500 ppm
Sep 25 10:02:37 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:37.967872 19891 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:37.968360 19891 webserver.cc:460] Webserver started at http://127.19.46.194:46521/ using document root <none> and password file <none>
I20260925 10:02:37.968600 19891 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:37.968672 19891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:37.968835 19891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:37.970007 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "c3c6e5e602c347baae87c91e18b91b95"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "e516be191961e7e5da37a246e2455e05"
server_key_iv: "502fefcc901c04304b4af6160abc439f"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.970394 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "c3c6e5e602c347baae87c91e18b91b95"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "e516be191961e7e5da37a246e2455e05"
server_key_iv: "502fefcc901c04304b4af6160abc439f"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.971830 19891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:37.972674 19906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:37.972900 19891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:37.973013 19891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "c3c6e5e602c347baae87c91e18b91b95"
format_stamp: "Formatted at 2026-09-25 10:02:37 on dist-test-slave-f9rl"
server_key: "e516be191961e7e5da37a246e2455e05"
server_key_iv: "502fefcc901c04304b4af6160abc439f"
server_key_version: "encryptionkey@0"
I20260925 10:02:37.973094 19891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:38.005052 19891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:38.007465 19891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:38.007632 19891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:38.007894 19891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:38.008219 19891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:38.008271 19891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:38.008312 19891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:38.008358 19891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:38.014431 19891 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:36091
I20260925 10:02:38.014477 20019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:36091 every 8 connection(s)
I20260925 10:02:38.014822 19891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330557, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:38.024755 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.015587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39581 (local address 127.19.46.254:36899)
0925 10:02:38.015662 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:38.015664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:38.015865 (+ 201us) server_negotiation.cc:407] Connection header received
0925 10:02:38.016422 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:38.016423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:38.016437 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:38.016487 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:38.017482 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.018004 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.019086 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.019368 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.019457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.021749 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:38.021751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:38.021751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:38.021758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:38.023146 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.023591 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.023593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.023593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.023662 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.023974 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.023975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.023976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.024102 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:38.024204 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:38.024481 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 10:02:38.024581 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:02:38.024817 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 19891
I20260925 10:02:38.024885 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:38.024935 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.015560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36899 (local address 127.19.46.194:39581)
0925 10:02:38.015746 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:38.015783 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:38.016305 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:38.016564 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:38.016568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:38.017155 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:38.017358 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:38.017362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.018106 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.018107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:38.018948 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:38.018950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.019466 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.019468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:38.019597 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.020192 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:38.020201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:38.021639 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:38.023249 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:38.023267 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:38.023270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:38.023497 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:38.023731 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:38.023732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:38.023732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:38.023899 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:38.024193 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:38.024199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:38.024347 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:02:38.024461 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:02:38.024684 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260925 10:02:38.025059 19643 external_mini_cluster.cc:1428] Setting key cf3c9433334bcdcff01d886cc86f742f
I20260925 10:02:38.025592 20020 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36899
I20260925 10:02:38.025799 20020 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:38.026026 20020 heartbeater.cc:511] Master 127.19.46.254:36899 requested a full tablet report, sending...
I20260925 10:02:38.026443 19695 ts_manager.cc:194] Registered new tserver with Master: c3c6e5e602c347baae87c91e18b91b95 (127.19.46.194:36091)
I20260925 10:02:38.026835 19695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:39581
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:38.057405 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:38.148528 20025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:38.148754 20025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:38.148782 20025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:38.150036 20025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:38.150091 20025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:38.150144 20025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:38.151696 20025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:38.151962 20025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:38.152220 20025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:38.157578 20031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:38.157644 20033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:38.157704 20025 server_base.cc:1034] running on GCE node
W20260925 10:02:38.157578 20030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:38.158021 20025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:38.158353 20025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:38.159528 20025 hybrid_clock.cc:648] HybridClock initialized: now 1790330558159514 us; error 42 us; skew 500 ppm
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:38.162652 20025 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:38.163204 20025 webserver.cc:460] Webserver started at http://127.19.46.195:40905/ using document root <none> and password file <none>
I20260925 10:02:38.163501 20025 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:38.163582 20025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:38.163730 20025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:38.164827 20025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "23e0050e70e448aebfa4ab55a0aa1fc3"
format_stamp: "Formatted at 2026-09-25 10:02:38 on dist-test-slave-f9rl"
server_key: "93b1527916d12ee271a2a099e1bba337"
server_key_iv: "084b002ce734768abf23da032a29b289"
server_key_version: "encryptionkey@0"
I20260925 10:02:38.165213 20025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "23e0050e70e448aebfa4ab55a0aa1fc3"
format_stamp: "Formatted at 2026-09-25 10:02:38 on dist-test-slave-f9rl"
server_key: "93b1527916d12ee271a2a099e1bba337"
server_key_iv: "084b002ce734768abf23da032a29b289"
server_key_version: "encryptionkey@0"
I20260925 10:02:38.166775 20025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:02:38.167876 20040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:38.168119 20025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:38.168237 20025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "23e0050e70e448aebfa4ab55a0aa1fc3"
format_stamp: "Formatted at 2026-09-25 10:02:38 on dist-test-slave-f9rl"
server_key: "93b1527916d12ee271a2a099e1bba337"
server_key_iv: "084b002ce734768abf23da032a29b289"
server_key_version: "encryptionkey@0"
I20260925 10:02:38.168340 20025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:38.180243 20025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:38.182876 20025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:38.183002 20025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:38.183190 20025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:38.183513 20025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:38.183549 20025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:38.183573 20025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:38.183624 20025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:38.189249 20025 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:38473
I20260925 10:02:38.189330 20153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:38473 every 8 connection(s)
I20260925 10:02:38.189709 20025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:38.194079 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20025
I20260925 10:02:38.194159 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:38.194339 19643 external_mini_cluster.cc:1428] Setting key b99b78533cfb04c85b888ab3cb91891d
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:38.200541 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.190459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55719 (local address 127.19.46.254:36899)
0925 10:02:38.190540 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:38.190542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:38.190760 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:02:38.191249 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:38.191251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:38.191261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:38.191328 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:38.192302 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.192823 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.194030 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.194300 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.194374 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.197003 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:38.197004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:38.197005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:38.197013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:38.198831 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.199505 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.199506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.199507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.199571 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.199897 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.199898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.199899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.200041 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:38.200108 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:38.200392 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 10:02:38.200467 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:02:38.200762 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.190420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36899 (local address 127.19.46.195:55719)
0925 10:02:38.190608 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:38.190646 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:38.191137 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:38.191398 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:38.191401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:38.191997 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:38.192201 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:38.192205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.192920 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.192921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:38.193929 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:38.193931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.194397 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.194398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:38.194545 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.195152 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:38.195160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:38.196838 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:38.199102 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:38.199111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:38.199113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:38.199396 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:38.199674 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:38.199675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:38.199675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:38.199798 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:38.200115 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:38.200119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:38.200254 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 10:02:38.200358 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:02:38.200505 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260925 10:02:38.201351 20154 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36899
I20260925 10:02:38.201488 20154 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:38.201699 20154 heartbeater.cc:511] Master 127.19.46.254:36899 requested a full tablet report, sending...
I20260925 10:02:38.202199 19695 ts_manager.cc:194] Registered new tserver with Master: 23e0050e70e448aebfa4ab55a0aa1fc3 (127.19.46.195:38473)
I20260925 10:02:38.202625 19695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:55719
I20260925 10:02:38.206683 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:38.233383 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.224157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.19.46.193:40667)
0925 10:02:38.224358 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:38.224363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:38.224376 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:02:38.224406 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:38.224409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:38.224425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:38.224502 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:38.225162 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.226016 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.226972 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.227351 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.227425 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.229942 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:38.229952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:38.229952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:38.229960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:38.231910 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.232315 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.232317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.232318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.232390 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.232620 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.232621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.232621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.232777 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:38.232889 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:38.233116 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:02:38.233198 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
W20260925 10:02:38.233899 19800 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37480
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[19650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:38.258980 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:38.250783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.19.46.193:40667)
0925 10:02:38.250881 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:38.250883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:38.250931 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:02:38.250994 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:38.250995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:38.251005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:38.251058 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:38.251753 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.252264 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.253101 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:38.253356 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:38.253418 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:38.256011 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:38.256012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:38.256012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:38.256021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:38.257630 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.258046 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.258047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.258048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.258113 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:38.258338 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:38.258339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:38.258340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:38.258462 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:38.258588 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:38.258759 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:02:38.258838 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:02:38.260123 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 19757
I20260925 10:02:38.266382 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 19891
I20260925 10:02:38.272334 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20025
I20260925 10:02:38.278095 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 19663
2026-09-25T10:02:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2002 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[20168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[20168](info): set up 2 sockets
Sep 25 10:02:38 dist-test-slave-f9rl krb5kdc[20168](info): commencing operation
krb5kdc: starting...
W20260925 10:02:39.335453 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:39.465655 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34011
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:34011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:39.552089 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:39.552353 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:39.552394 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:39.553617 20181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:39.553687 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:39.553712 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:39.553740 20181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:39.553771 20181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:39.555341 20181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:34011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:39.555646 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:39.555899 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:39.561239 20187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:39.561259 20186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:39.561414 20181 server_base.cc:1034] running on GCE node
W20260925 10:02:39.561266 20189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:39.561807 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:39.562134 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:39.563344 20181 hybrid_clock.cc:648] HybridClock initialized: now 1790330559563320 us; error 40 us; skew 500 ppm
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:39.566198 20181 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:39.566720 20181 webserver.cc:460] Webserver started at http://127.19.46.254:40959/ using document root <none> and password file <none>
I20260925 10:02:39.566974 20181 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:39.567025 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:39.567220 20181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:39.568447 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "91351078d6ab47b28d06570a94c44a8c"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "657011a6f3c02de94a80b24028b041cd"
server_key_iv: "93bae1521a680983e1fac8902ca57c34"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.568864 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "91351078d6ab47b28d06570a94c44a8c"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "657011a6f3c02de94a80b24028b041cd"
server_key_iv: "93bae1521a680983e1fac8902ca57c34"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.570459 20181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:39.571276 20196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.571460 20181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:39.571583 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "91351078d6ab47b28d06570a94c44a8c"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "657011a6f3c02de94a80b24028b041cd"
server_key_iv: "93bae1521a680983e1fac8902ca57c34"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.571667 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:39.593796 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:39.596429 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:39.596637 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:39.601346 20181 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:34011
I20260925 10:02:39.601404 20248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:34011 every 8 connection(s)
I20260925 10:02:39.601837 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:39.602833 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:39.605149 20249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c: Bootstrap starting.
I20260925 10:02:39.605803 20249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:39.606045 20249 log.cc:824] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:39.606748 20249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c: No bootstrap required, opened a new log
I20260925 10:02:39.608371 20249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } }
I20260925 10:02:39.608520 20249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:39.608551 20249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91351078d6ab47b28d06570a94c44a8c, State: Initialized, Role: FOLLOWER
I20260925 10:02:39.608697 20249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } }
I20260925 10:02:39.608780 20249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:39.608807 20249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:39.608839 20249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:39.609359 20249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } }
I20260925 10:02:39.609508 20249 leader_election.cc:302] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91351078d6ab47b28d06570a94c44a8c; no voters:
I20260925 10:02:39.609719 20249 leader_election.cc:288] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:39.609833 20252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:39.610000 20252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [term 1 LEADER]: Becoming Leader. State: Replica: 91351078d6ab47b28d06570a94c44a8c, State: Running, Role: LEADER
I20260925 10:02:39.610036 20249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:39.610103 20252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } }
I20260925 10:02:39.610571 20254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91351078d6ab47b28d06570a94c44a8c. Latest consensus state: current_term: 1 leader_uuid: "91351078d6ab47b28d06570a94c44a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } } }
I20260925 10:02:39.610667 20254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:39.610728 20253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91351078d6ab47b28d06570a94c44a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91351078d6ab47b28d06570a94c44a8c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34011 } } }
I20260925 10:02:39.610800 20253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:39.611334 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20181
I20260925 10:02:39.611425 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:39.611415 20261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:39.611611 19643 external_mini_cluster.cc:1428] Setting key 4f5a3b8cd9ea07c360aa986a029a6be7
I20260925 10:02:39.611964 20261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:39.613655 20261 catalog_manager.cc:1347] Generated new cluster ID: b66da0acbe9d41a7b5eb64fdab6e772a
I20260925 10:02:39.613709 20261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:39.623733 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:39.612138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.19.46.254:34011)
0925 10:02:39.612752 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:39.612759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:39.612782 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:02:39.612907 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:39.612915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:39.612986 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:39.613151 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:39.613905 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.615361 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.616435 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.617083 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.617213 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:39.619998 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:39.620003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:39.620003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:39.620016 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:39.621992 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.622446 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.622449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.622450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.622561 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.622774 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.622774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.622775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.623081 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:39.623220 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:39.623361 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 10:02:39.623490 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":65,"threads_started":1}
I20260925 10:02:39.630317 20261 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:39.630904 20261 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:39.639278 20261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91351078d6ab47b28d06570a94c44a8c: Generated new TSK 0
I20260925 10:02:39.639515 20261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:39.668431 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:39.752271 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:39.752503 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:39.752569 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:39.753837 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:39.753918 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:39.753991 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:39.755589 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:39.755867 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:39.756115 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:02:39.761669 20275 server_base.cc:1034] running on GCE node
W20260925 10:02:39.761637 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:39.761641 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:39.761647 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:39.762022 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:39.762318 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:39.763474 20275 hybrid_clock.cc:648] HybridClock initialized: now 1790330559763459 us; error 35 us; skew 500 ppm
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:39.766183 20275 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:39.766641 20275 webserver.cc:460] Webserver started at http://127.19.46.193:35457/ using document root <none> and password file <none>
I20260925 10:02:39.766873 20275 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:39.766981 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:39.767144 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:39.768262 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "52d9c7c005054da58be4a8eb66afbfe4"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "d717d545a62313c83a23770b2a8392e3"
server_key_iv: "0c2c0a93939c12f12c33483202209867"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.768638 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "52d9c7c005054da58be4a8eb66afbfe4"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "d717d545a62313c83a23770b2a8392e3"
server_key_iv: "0c2c0a93939c12f12c33483202209867"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.770154 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:39.770972 20290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.771164 20275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:39.771270 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "52d9c7c005054da58be4a8eb66afbfe4"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "d717d545a62313c83a23770b2a8392e3"
server_key_iv: "0c2c0a93939c12f12c33483202209867"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.771356 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:39.792981 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:39.795469 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:39.795651 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:39.795907 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:39.796228 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:39.796284 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.796325 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:39.796368 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.802361 20275 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:38345
I20260925 10:02:39.802407 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:38345 every 8 connection(s)
I20260925 10:02:39.802732 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:39.803566 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20275
I20260925 10:02:39.803665 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:39.803862 19643 external_mini_cluster.cc:1428] Setting key fd3dff6f8c0939e210095d2100a9b8c9
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:02:39.813424 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:39.803498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:44309 (local address 127.19.46.254:34011)
0925 10:02:39.803592 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:39.803594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:39.803839 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:02:39.805029 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:39.805031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:39.805044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:39.805115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:39.805925 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.806419 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.807460 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.807675 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.807788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:39.810377 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:39.810378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:39.810378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:39.810385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:39.812071 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.812454 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.812455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.812455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.812505 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.812766 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.812767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.812767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.812941 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:39.813017 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:39.813285 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 10:02:39.813355 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:02:39.813655 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:39.803421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:34011 (local address 127.19.46.193:44309)
0925 10:02:39.803698 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:39.803742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:39.804913 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:39.805188 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:39.805192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:39.805508 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:39.805794 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:39.805802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.806518 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.806519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:39.807349 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:39.807351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.807813 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.807813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:39.807943 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:39.808492 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:39.808501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:39.810282 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:39.812157 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:39.812164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:39.812166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:39.812386 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:39.812566 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:39.812567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:39.812567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:39.812667 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:39.813014 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:39.813018 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:39.813163 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:02:39.813250 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:02:39.813376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260925 10:02:39.814210 20404 heartbeater.cc:347] Connected to a master server at 127.19.46.254:34011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
I20260925 10:02:39.814339 20404 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:39.814545 20404 heartbeater.cc:511] Master 127.19.46.254:34011 requested a full tablet report, sending...
I20260925 10:02:39.815133 20213 ts_manager.cc:194] Registered new tserver with Master: 52d9c7c005054da58be4a8eb66afbfe4 (127.19.46.193:38345)
I20260925 10:02:39.816051 20213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:44309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:39.830543 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:39.911820 20409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:39.912034 20409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:39.912060 20409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:39.913185 20409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:39.913220 20409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:39.913247 20409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:39.914764 20409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:39.914952 20409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:39.915132 20409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:39.920343 20417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:39.920360 20415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:39.920364 20414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:39.920392 20409 server_base.cc:1034] running on GCE node
I20260925 10:02:39.920751 20409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:39.921020 20409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:39.922186 20409 hybrid_clock.cc:648] HybridClock initialized: now 1790330559922166 us; error 38 us; skew 500 ppm
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:39.924892 20409 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:39.925380 20409 webserver.cc:460] Webserver started at http://127.19.46.194:39013/ using document root <none> and password file <none>
I20260925 10:02:39.925639 20409 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:39.925710 20409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:39.925860 20409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:39.926935 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "b08d8afb4da14f43bcf9d9acc75d3d34"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "4636388fd9b9fd89ed345b3da35d591a"
server_key_iv: "a9a8e988ad9fe5e4eed39f18a7aa90e9"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.927314 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "b08d8afb4da14f43bcf9d9acc75d3d34"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "4636388fd9b9fd89ed345b3da35d591a"
server_key_iv: "a9a8e988ad9fe5e4eed39f18a7aa90e9"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.928754 20409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:39.929621 20424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.929839 20409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:39.929960 20409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "b08d8afb4da14f43bcf9d9acc75d3d34"
format_stamp: "Formatted at 2026-09-25 10:02:39 on dist-test-slave-f9rl"
server_key: "4636388fd9b9fd89ed345b3da35d591a"
server_key_iv: "a9a8e988ad9fe5e4eed39f18a7aa90e9"
server_key_version: "encryptionkey@0"
I20260925 10:02:39.930048 20409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:39.941735 20409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:39.944022 20409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:39.944208 20409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:39.944455 20409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:39.944792 20409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:39.944844 20409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.944888 20409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:39.944911 20409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:39.951042 20409 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:38983
I20260925 10:02:39.951104 20537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:38983 every 8 connection(s)
I20260925 10:02:39.951460 20409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:02:39.955294 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20409
I20260925 10:02:39.955365 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:39.955538 19643 external_mini_cluster.cc:1428] Setting key 6c1c12a5f393d7a3c71e711789777330
Sep 25 10:02:39 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:39.962191 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:39.952352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:56861 (local address 127.19.46.254:34011)
0925 10:02:39.952400 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:39.952402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:39.952614 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:02:39.953740 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:39.953741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:39.953754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:39.953818 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:39.954469 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.954985 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.956327 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.956532 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.956615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:39.959218 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:39.959219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:39.959220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:39.959229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:39.960707 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.961143 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.961145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.961145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.961205 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:39.961508 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:39.961509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:39.961509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:39.961643 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:39.961723 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:39.962008 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:02:39.962095 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:02:39.962406 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:39.952317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:34011 (local address 127.19.46.194:56861)
0925 10:02:39.952494 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:39.952529 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:39.953625 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:39.953896 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:39.953900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:39.954178 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:39.954371 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:39.954375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.955095 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.955096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:39.956238 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:39.956241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:39.956625 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:39.956626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:39.956764 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:39.957413 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:39.957426 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:39.959110 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:39.960797 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:39.960804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:39.960807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:39.961040 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:39.961274 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:39.961275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:39.961276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:39.961395 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:39.961714 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:39.961721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:39.961869 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:02:39.961973 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:02:39.962150 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260925 10:02:39.962917 20538 heartbeater.cc:347] Connected to a master server at 127.19.46.254:34011
I20260925 10:02:39.963049 20538 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:39.963265 20538 heartbeater.cc:511] Master 127.19.46.254:34011 requested a full tablet report, sending...
I20260925 10:02:39.963717 20213 ts_manager.cc:194] Registered new tserver with Master: b08d8afb4da14f43bcf9d9acc75d3d34 (127.19.46.194:38983)
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:02:39.964260 20213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:56861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:39.984478 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:40.068593 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:40.068856 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:40.068883 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:40.070340 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:40.070402 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:40.070436 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:40.072104 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:40.072396 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:40.072676 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:40.078217 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:40.078230 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:40.078235 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:40.078353 20543 server_base.cc:1034] running on GCE node
I20260925 10:02:40.078723 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:40.078989 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:40.080214 20543 hybrid_clock.cc:648] HybridClock initialized: now 1790330560080192 us; error 37 us; skew 500 ppm
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:40.083218 20543 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:40.083741 20543 webserver.cc:460] Webserver started at http://127.19.46.195:43913/ using document root <none> and password file <none>
I20260925 10:02:40.084056 20543 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:40.084142 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:40.084293 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:40.085371 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "4de88d4907ae489c992b3c4df451cc5f"
format_stamp: "Formatted at 2026-09-25 10:02:40 on dist-test-slave-f9rl"
server_key: "78a11e1a8d43dbb2146262f99ce2b19e"
server_key_iv: "582486ba7ac828e2cb0a08115c647d62"
server_key_version: "encryptionkey@0"
I20260925 10:02:40.085821 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "4de88d4907ae489c992b3c4df451cc5f"
format_stamp: "Formatted at 2026-09-25 10:02:40 on dist-test-slave-f9rl"
server_key: "78a11e1a8d43dbb2146262f99ce2b19e"
server_key_iv: "582486ba7ac828e2cb0a08115c647d62"
server_key_version: "encryptionkey@0"
I20260925 10:02:40.087217 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:40.087993 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:40.088224 20543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:40.088328 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "4de88d4907ae489c992b3c4df451cc5f"
format_stamp: "Formatted at 2026-09-25 10:02:40 on dist-test-slave-f9rl"
server_key: "78a11e1a8d43dbb2146262f99ce2b19e"
server_key_iv: "582486ba7ac828e2cb0a08115c647d62"
server_key_version: "encryptionkey@0"
I20260925 10:02:40.088428 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:40.099558 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:40.102030 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:40.102210 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:40.102454 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:40.102778 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:40.102836 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:40.102876 20543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:40.102905 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:40.108980 20543 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:37397
I20260925 10:02:40.109030 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:37397 every 8 connection(s)
I20260925 10:02:40.109378 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:40.109606 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20543
I20260925 10:02:40.109728 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:40.109928 19643 external_mini_cluster.cc:1428] Setting key 528b3430a769f1983e4848d3b6c89bb4
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:40.120079 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.110271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:44545 (local address 127.19.46.254:34011)
0925 10:02:40.110396 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.110398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.110691 (+ 293us) server_negotiation.cc:407] Connection header received
0925 10:02:40.111807 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.111808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.111821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.111876 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:40.112536 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.113033 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.114136 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.114330 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.114435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.116799 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:40.116801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:40.116801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:40.116809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:40.118595 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.119069 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.119071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.119071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.119135 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.119400 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.119401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.119401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.119529 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:40.119607 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.119894 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.119982 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 10:02:40.120302 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.110203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:34011 (local address 127.19.46.195:44545)
0925 10:02:40.110514 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.110578 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.111665 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.111950 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.111954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.112235 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:40.112429 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.112433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.113133 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.113134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.114034 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.114036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.114443 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.114444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.114574 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.115136 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:40.115145 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:40.116692 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:40.118723 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:40.118731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:40.118734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:40.118973 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:40.119211 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:40.119212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:40.119212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:40.119332 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:40.119597 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:40.119603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:40.119756 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.119871 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.120041 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
I20260925 10:02:40.120926 20672 heartbeater.cc:347] Connected to a master server at 127.19.46.254:34011
I20260925 10:02:40.121042 20672 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:40.121295 20672 heartbeater.cc:511] Master 127.19.46.254:34011 requested a full tablet report, sending...
I20260925 10:02:40.121784 20213 ts_manager.cc:194] Registered new tserver with Master: 4de88d4907ae489c992b3c4df451cc5f (127.19.46.195:37397)
I20260925 10:02:40.122042 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:02:40.122311 20213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:44545
I20260925 10:02:40.130614 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.123829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.19.46.254:34011)
0925 10:02:40.123873 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.123876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.123998 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:02:40.124068 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.124069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.124079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.124133 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:40.124675 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.125308 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.126328 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.126539 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.126633 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.127928 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:40.127930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:40.127931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:40.127938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:40.129361 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.129780 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.129782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.129782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.129841 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.130047 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.130048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.130048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.130212 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:40.130318 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.130430 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.130508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:02:40.133339 20213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:40.135147 20213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:40.142575 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.138145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.19.46.194:38983)
0925 10:02:40.138312 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.138318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.138329 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:40.138438 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.138441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.138459 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.138541 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:40.138759 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.140009 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.141324 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142108 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142199 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142271 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.142430 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.142475 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260925 10:02:40.142598 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.137897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:38983 (local address 127.0.0.1:37488)
0925 10:02:40.138279 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.138289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.138336 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.138579 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.138583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.138585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:40.138663 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.138666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.140151 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.140152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.141192 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.141196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142210 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.142306 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142322 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.142398 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.142486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":67,"thread_start_us":55,"threads_started":1}
I20260925 10:02:40.142598 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.137882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38345 (local address 127.0.0.1:33414)
0925 10:02:40.138001 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.138045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.138125 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.138856 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.138857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.138859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:40.138917 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.138919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.139917 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.139919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.141340 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.141344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142262 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.142338 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142342 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.142368 (+ 26us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.142490 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260925 10:02:40.142841 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.138092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.19.46.193:38345)
0925 10:02:40.138671 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.138676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.138687 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:40.138729 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.138731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.138745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.138996 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:40.139076 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.139826 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.141451 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142165 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142545 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142619 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.142646 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.142728 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":97,"threads_started":1}
I20260925 10:02:40.142880 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.138135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.19.46.195:37397)
0925 10:02:40.138418 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.138422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.138432 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:40.138477 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.138478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.138491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.138605 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:40.138811 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.140031 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.141561 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142182 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142251 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142312 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.142732 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.142783 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":92,"threads_started":1}
I20260925 10:02:40.142937 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.137897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:37397 (local address 127.0.0.1:41152)
0925 10:02:40.138307 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.138318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.138395 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.138694 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.138695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.138697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:40.138734 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.138736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.140149 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.140150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.141398 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.141400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.142344 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.142345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.142407 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.142411 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.142460 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.142493 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":56,"thread_start_us":53,"threads_started":1}
I20260925 10:02:40.143658 20338 tablet_service.cc:1467] Processing CreateTablet for tablet 903780c6a2d740fc9d305641253a745a (DEFAULT_TABLE table=test-table [id=2fd864371a4c487abe9663dfe3f864fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:40.143807 20472 tablet_service.cc:1467] Processing CreateTablet for tablet 903780c6a2d740fc9d305641253a745a (DEFAULT_TABLE table=test-table [id=2fd864371a4c487abe9663dfe3f864fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:40.144168 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903780c6a2d740fc9d305641253a745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:40.144170 20338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903780c6a2d740fc9d305641253a745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:40.146910 20687 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Bootstrap starting.
I20260925 10:02:40.147037 20686 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Bootstrap starting.
I20260925 10:02:40.147465 20606 tablet_service.cc:1467] Processing CreateTablet for tablet 903780c6a2d740fc9d305641253a745a (DEFAULT_TABLE table=test-table [id=2fd864371a4c487abe9663dfe3f864fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:40.147644 20687 tablet_bootstrap.cc:654] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:40.147727 20686 tablet_bootstrap.cc:654] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:40.147845 20606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903780c6a2d740fc9d305641253a745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:40.148218 20687 log.cc:824] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:40.148269 20686 log.cc:824] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:40.149103 20687 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: No bootstrap required, opened a new log
I20260925 10:02:40.149180 20687 ts_tablet_manager.cc:1397] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:02:40.149307 20686 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: No bootstrap required, opened a new log
I20260925 10:02:40.149395 20686 ts_tablet_manager.cc:1397] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:02:40.150787 20686 raft_consensus.cc:348] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.150961 20686 raft_consensus.cc:374] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:40.151006 20686 raft_consensus.cc:729] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08d8afb4da14f43bcf9d9acc75d3d34, State: Initialized, Role: FOLLOWER
I20260925 10:02:40.151139 20686 consensus_queue.cc:260] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.151437 20686 ts_tablet_manager.cc:1428] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:02:40.151324 20687 raft_consensus.cc:348] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.151494 20687 raft_consensus.cc:374] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:40.151557 20687 raft_consensus.cc:729] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d9c7c005054da58be4a8eb66afbfe4, State: Initialized, Role: FOLLOWER
I20260925 10:02:40.151572 20538 heartbeater.cc:503] Master 127.19.46.254:34011 was elected leader, sending a full tablet report...
I20260925 10:02:40.151674 20687 consensus_queue.cc:260] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.151444 20690 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Bootstrap starting.
I20260925 10:02:40.151932 20687 ts_tablet_manager.cc:1428] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:02:40.151932 20404 heartbeater.cc:503] Master 127.19.46.254:34011 was elected leader, sending a full tablet report...
I20260925 10:02:40.152630 20690 tablet_bootstrap.cc:654] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:40.153195 20690 log.cc:824] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:40.153983 20690 tablet_bootstrap.cc:492] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: No bootstrap required, opened a new log
I20260925 10:02:40.154042 20690 ts_tablet_manager.cc:1397] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:02:40.155340 20690 raft_consensus.cc:348] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.155476 20690 raft_consensus.cc:374] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:40.155498 20690 raft_consensus.cc:729] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de88d4907ae489c992b3c4df451cc5f, State: Initialized, Role: FOLLOWER
I20260925 10:02:40.155616 20690 consensus_queue.cc:260] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.155853 20690 ts_tablet_manager.cc:1428] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:02:40.155901 20672 heartbeater.cc:503] Master 127.19.46.254:34011 was elected leader, sending a full tablet report...
W20260925 10:02:40.202276 20539 tablet.cc:2369] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:40.296620 20692 raft_consensus.cc:482] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:40.296756 20692 raft_consensus.cc:504] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.297137 20692 leader_election.cc:288] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b08d8afb4da14f43bcf9d9acc75d3d34 (127.19.46.194:38983), 4de88d4907ae489c992b3c4df451cc5f (127.19.46.195:37397)
I20260925 10:02:40.301455 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.297453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:48027 (local address 127.19.46.194:38983)
0925 10:02:40.297508 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.297511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.297521 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:40.297559 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.297560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.297568 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.297662 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:40.298053 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.298744 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.300340 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.300968 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.301094 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.301103 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.301311 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.301359 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:02:40.301468 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.297466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:40265 (local address 127.19.46.195:37397)
0925 10:02:40.297509 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.297511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.297636 (+ 125us) server_negotiation.cc:407] Connection header received
0925 10:02:40.297664 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.297665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.297674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.297725 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:40.297996 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.298755 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.300340 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.300968 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.301060 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.301073 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.301310 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.301359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:02:40.301462 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.297338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:38983 (local address 127.19.46.193:48027)
0925 10:02:40.297405 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.297417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.297483 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.297691 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.297691 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.297693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:40.297794 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.297796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.298869 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.298870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.300186 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.300191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.301088 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.301090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.301205 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.301208 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.301263 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.301313 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":104}
I20260925 10:02:40.301472 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.297338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:37397 (local address 127.19.46.193:40265)
0925 10:02:40.297534 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:40.297543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:40.297598 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:40.297743 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:40.297744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:40.297746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:40.297807 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.297809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.298869 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.298870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.300186 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:40.300191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.301088 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.301090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:40.301179 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.301186 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:02:40.301260 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:02:40.301313 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":61,"thread_start_us":108,"threads_started":1}
I20260925 10:02:40.301959 20492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903780c6a2d740fc9d305641253a745a" candidate_uuid: "52d9c7c005054da58be4a8eb66afbfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" is_pre_election: true
I20260925 10:02:40.301986 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903780c6a2d740fc9d305641253a745a" candidate_uuid: "52d9c7c005054da58be4a8eb66afbfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de88d4907ae489c992b3c4df451cc5f" is_pre_election: true
I20260925 10:02:40.302134 20626 raft_consensus.cc:2454] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d9c7c005054da58be4a8eb66afbfe4 in term 0.
I20260925 10:02:40.302134 20492 raft_consensus.cc:2454] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d9c7c005054da58be4a8eb66afbfe4 in term 0.
I20260925 10:02:40.302419 20293 leader_election.cc:302] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d9c7c005054da58be4a8eb66afbfe4, b08d8afb4da14f43bcf9d9acc75d3d34; no voters:
I20260925 10:02:40.302585 20692 raft_consensus.cc:2781] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:40.302664 20692 raft_consensus.cc:482] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:40.302683 20692 raft_consensus.cc:3037] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:02:40.303364 20405 tablet.cc:2369] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:40.303401 20692 raft_consensus.cc:504] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.303563 20692 leader_election.cc:288] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [CANDIDATE]: Term 1 election: Requested vote from peers b08d8afb4da14f43bcf9d9acc75d3d34 (127.19.46.194:38983), 4de88d4907ae489c992b3c4df451cc5f (127.19.46.195:37397)
I20260925 10:02:40.303709 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903780c6a2d740fc9d305641253a745a" candidate_uuid: "52d9c7c005054da58be4a8eb66afbfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de88d4907ae489c992b3c4df451cc5f"
I20260925 10:02:40.303746 20492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903780c6a2d740fc9d305641253a745a" candidate_uuid: "52d9c7c005054da58be4a8eb66afbfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34"
I20260925 10:02:40.303814 20626 raft_consensus.cc:3037] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:40.303817 20492 raft_consensus.cc:3037] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:40.304519 20626 raft_consensus.cc:2454] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d9c7c005054da58be4a8eb66afbfe4 in term 1.
I20260925 10:02:40.304561 20492 raft_consensus.cc:2454] T 903780c6a2d740fc9d305641253a745a P b08d8afb4da14f43bcf9d9acc75d3d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d9c7c005054da58be4a8eb66afbfe4 in term 1.
I20260925 10:02:40.304704 20294 leader_election.cc:302] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de88d4907ae489c992b3c4df451cc5f, 52d9c7c005054da58be4a8eb66afbfe4; no voters:
I20260925 10:02:40.304801 20692 raft_consensus.cc:2781] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:40.304991 20692 raft_consensus.cc:686] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [term 1 LEADER]: Becoming Leader. State: Replica: 52d9c7c005054da58be4a8eb66afbfe4, State: Running, Role: LEADER
I20260925 10:02:40.305109 20692 consensus_queue.cc:237] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } }
I20260925 10:02:40.305998 20213 catalog_manager.cc:5495] T 903780c6a2d740fc9d305641253a745a P 52d9c7c005054da58be4a8eb66afbfe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52d9c7c005054da58be4a8eb66afbfe4 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "52d9c7c005054da58be4a8eb66afbfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9c7c005054da58be4a8eb66afbfe4" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b08d8afb4da14f43bcf9d9acc75d3d34" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4de88d4907ae489c992b3c4df451cc5f" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 37397 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:40.334903 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.325960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.19.46.193:38345)
0925 10:02:40.326077 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.326079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.326094 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:02:40.326181 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.326182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.326191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.326240 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:40.326817 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.327350 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.328176 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.328373 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.328445 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.331517 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:40.331533 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:40.331533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:40.331541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:40.333561 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.333990 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.333994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.333994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.334079 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.334295 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.334296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.334297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.334466 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:40.334634 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.334700 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.334792 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:40.359754 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.351827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.19.46.193:38345)
0925 10:02:40.351903 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.351905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.351961 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:02:40.352015 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.352016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.352025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.352084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:40.352724 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.353245 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.354068 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.354275 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.354341 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.356700 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:40.356701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:40.356702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:40.356709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:40.358425 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.358809 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.358810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.358811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.358874 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.359144 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.359146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.359146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.359262 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:40.359352 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.359610 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.359682 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 10:02:40.360332 20673 tablet.cc:2369] T 903780c6a2d740fc9d305641253a745a P 4de88d4907ae489c992b3c4df451cc5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:40.386339 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:40.377093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.19.46.193:38345)
0925 10:02:40.377222 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:40.377224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:40.377236 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:02:40.377303 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:40.377304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:40.377314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:40.377378 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:40.378108 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.378611 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.379565 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:40.379768 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:40.379843 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:40.382383 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:40.382385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:40.382385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:40.382393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:40.384271 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.384892 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.384894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.384895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.385000 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:40.385348 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:40.385351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:40.385352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:40.385590 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:40.385706 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:40.386019 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 10:02:40.386143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 10:02:40.387192 20318 tablet_service.cc:2908] Rejecting scan request for tablet 903780c6a2d740fc9d305641253a745a: Uninitialized: clean time has not yet been initialized
I20260925 10:02:40.388736 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20275
I20260925 10:02:40.395112 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20409
I20260925 10:02:40.402555 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20543
I20260925 10:02:40.408756 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20181
2026-09-25T10:02:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2131 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20710](info): set up 2 sockets
Sep 25 10:02:40 dist-test-slave-f9rl krb5kdc[20710](info): commencing operation
krb5kdc: starting...
W20260925 10:02:41.490644 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:41 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:41.624759 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46317
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:41.711891 20723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:41.712172 20723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:41.712213 20723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:41.713546 20723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:41.713624 20723 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:02:41.713646 20723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:41.713667 20723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:41.713701 20723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:41.713730 20723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:41.716215 20723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46317
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:41.716740 20723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:41.717072 20723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:41.723488 20728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:41.723464 20731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:41.723450 20729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:41.723866 20723 server_base.cc:1034] running on GCE node
I20260925 10:02:41.724184 20723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:41.724546 20723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:41.725757 20723 hybrid_clock.cc:648] HybridClock initialized: now 1790330561725717 us; error 53 us; skew 500 ppm
Sep 25 10:02:41 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:41.735985 20723 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:41.736667 20723 webserver.cc:460] Webserver started at http://127.19.46.254:37193/ using document root <none> and password file <none>
I20260925 10:02:41.736994 20723 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:41.737051 20723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:41.737210 20723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:41.738672 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "e380e4e2d79c46ec852e75a65641cc70"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "b91ee488f8bcfc2531b05810f4472238"
server_key_iv: "20031a6589566adcb4a8f85644205df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.739168 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "e380e4e2d79c46ec852e75a65641cc70"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "b91ee488f8bcfc2531b05810f4472238"
server_key_iv: "20031a6589566adcb4a8f85644205df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.741343 20723 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:02:41.742583 20738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:41.742980 20723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:41.743142 20723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "e380e4e2d79c46ec852e75a65641cc70"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "b91ee488f8bcfc2531b05810f4472238"
server_key_iv: "20031a6589566adcb4a8f85644205df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.743254 20723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:41.787423 20723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:41.790560 20723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:41.790813 20723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:41.799088 20723 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46317
I20260925 10:02:41.799237 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46317 every 8 connection(s)
I20260925 10:02:41.800030 20723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:41.801179 20801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:41.801748 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20723
I20260925 10:02:41.802035 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:41.802299 19643 external_mini_cluster.cc:1428] Setting key 9334cea2d296d60f1b9a723ade6d0812
I20260925 10:02:41.804591 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Bootstrap starting.
I20260925 10:02:41.805219 20801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:41.805612 20801 log.cc:824] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:41.806401 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: No bootstrap required, opened a new log
I20260925 10:02:41.808372 20801 raft_consensus.cc:348] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } }
I20260925 10:02:41.808573 20801 raft_consensus.cc:374] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:41.808627 20801 raft_consensus.cc:729] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e380e4e2d79c46ec852e75a65641cc70, State: Initialized, Role: FOLLOWER
I20260925 10:02:41.808780 20801 consensus_queue.cc:260] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } }
I20260925 10:02:41.808881 20801 raft_consensus.cc:388] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:41.808926 20801 raft_consensus.cc:482] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:41.808976 20801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:41.809854 20801 raft_consensus.cc:504] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } }
I20260925 10:02:41.810002 20801 leader_election.cc:302] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e380e4e2d79c46ec852e75a65641cc70; no voters:
I20260925 10:02:41.810230 20801 leader_election.cc:288] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:41.810302 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:41.810534 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [term 1 LEADER]: Becoming Leader. State: Replica: e380e4e2d79c46ec852e75a65641cc70, State: Running, Role: LEADER
I20260925 10:02:41.810541 20801 sys_catalog.cc:564] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:41.810694 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } }
I20260925 10:02:41.811249 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e380e4e2d79c46ec852e75a65641cc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } } }
I20260925 10:02:41.811343 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [sys.catalog]: This master's current role is: LEADER
Sep 25 10:02:41 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:41.812887 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e380e4e2d79c46ec852e75a65641cc70. Latest consensus state: current_term: 1 leader_uuid: "e380e4e2d79c46ec852e75a65641cc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e380e4e2d79c46ec852e75a65641cc70" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46317 } } }
I20260925 10:02:41.812990 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70 [sys.catalog]: This master's current role is: LEADER
W20260925 10:02:41.813367 20822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:02:41.813424 20822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:02:41.813540 20810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:41.813730 20810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:41.815557 20810 catalog_manager.cc:1347] Generated new cluster ID: 95b0629f1c9a4cf18795f3b8b955077a
I20260925 10:02:41.815614 20810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:02:41.816854 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:41.803112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.19.46.254:46317)
0925 10:02:41.803342 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:41.803352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:41.805016 (+ 1664us) server_negotiation.cc:407] Connection header received
0925 10:02:41.805643 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:41.805652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:41.805754 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:41.805915 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:41.806617 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:41.807557 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:41.808358 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:41.808929 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:41.809053 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:41.812601 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:41.812606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:41.812606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:41.812621 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:41.814890 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:41.815385 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:41.815390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:41.815390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:41.815523 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:41.815775 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:41.815776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:41.815776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:41.816120 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:41.816294 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:41.816409 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:02:41.816616 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":128,"threads_started":1}
I20260925 10:02:41.827668 20810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:41.828595 20810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:41.838367 20810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Generated new TSK 0
I20260925 10:02:41.838662 20810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:41.873867 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46317
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:41.966017 20827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:41.966300 20827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:41.966343 20827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:41.967599 20827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:41.967679 20827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:41.967743 20827 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:41.967799 20827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:41.969393 20827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:41.969687 20827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:41.969897 20827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:41.975564 20835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:41.975600 20832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:41.975564 20833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:41.975859 20827 server_base.cc:1034] running on GCE node
I20260925 10:02:41.976126 20827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:41.976402 20827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:41.977576 20827 hybrid_clock.cc:648] HybridClock initialized: now 1790330561977533 us; error 71 us; skew 500 ppm
Sep 25 10:02:41 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:41.980480 20827 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:41.981019 20827 webserver.cc:460] Webserver started at http://127.19.46.193:34001/ using document root <none> and password file <none>
I20260925 10:02:41.981292 20827 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:41.981341 20827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:41.981513 20827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:41.982703 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "390a90a13ed542bf8cc91b1c1a121ac2"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "7b3c69316ca82175215149a01338ec3f"
server_key_iv: "3911aa293b8eaaaae466ccff89846deb"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.983094 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "390a90a13ed542bf8cc91b1c1a121ac2"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "7b3c69316ca82175215149a01338ec3f"
server_key_iv: "3911aa293b8eaaaae466ccff89846deb"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.984606 20827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:41.985839 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:41.986268 20827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:41.986368 20827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "390a90a13ed542bf8cc91b1c1a121ac2"
format_stamp: "Formatted at 2026-09-25 10:02:41 on dist-test-slave-f9rl"
server_key: "7b3c69316ca82175215149a01338ec3f"
server_key_iv: "3911aa293b8eaaaae466ccff89846deb"
server_key_version: "encryptionkey@0"
I20260925 10:02:41.986502 20827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:42.001380 20827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:42.004120 20827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:42.004316 20827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:42.005021 20827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:42.005075 20827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.005105 20827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:42.005118 20827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.013691 20827 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:43045
I20260925 10:02:42.013746 20957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:43045 every 8 connection(s)
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:42.014222 20827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:42.017964 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.005549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:50101 (local address 127.19.46.254:46317)
0925 10:02:42.005607 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.005610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.005942 (+ 332us) server_negotiation.cc:407] Connection header received
0925 10:02:42.006813 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.006815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.006829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.006902 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.008062 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.008584 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.010614 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.010913 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.010997 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.014462 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.014465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.014466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.014475 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.015983 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.016562 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.016563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.016564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.016625 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.017025 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.017026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.017027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.017193 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.017261 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.017757 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.017875 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:02:42.018064 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.005558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.193:50101)
0925 10:02:42.005782 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.005846 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.006668 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.007032 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.007039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.007532 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.007933 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.007942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.008705 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.008706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.010482 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.010486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.011084 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.011086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.011308 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.012064 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.012076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.014339 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.016095 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.016103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.016106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.016398 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.016756 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.016757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.016758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.016896 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.017288 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.017295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.017584 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.017684 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.017837 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260925 10:02:42.018708 20958 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46317
I20260925 10:02:42.018854 20958 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:42.019091 20958 heartbeater.cc:511] Master 127.19.46.254:46317 requested a full tablet report, sending...
I20260925 10:02:42.019822 20754 ts_manager.cc:194] Registered new tserver with Master: 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193:43045)
I20260925 10:02:42.020607 20754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:50101
I20260925 10:02:42.021129 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20827
I20260925 10:02:42.021240 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:42.021411 19643 external_mini_cluster.cc:1428] Setting key 5116431b46820b5f0b7b638a3912c615
I20260925 10:02:42.028833 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.019791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.193:53351)
0925 10:02:42.020886 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.020898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.020986 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.021866 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.021867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.022095 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.022168 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.022171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.022851 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.022853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.023390 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.023391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.023791 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.023792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.023871 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.024482 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.024488 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.025064 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.027283 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.027284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.027285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.027473 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.027764 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.027764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.027765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.027826 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.028317 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.028319 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.028432 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.028546 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.028703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":98,"threads_started":1}
I20260925 10:02:42.028894 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.020101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:53351 (local address 127.19.46.254:46317)
0925 10:02:42.020185 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.020187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.020957 (+ 770us) server_negotiation.cc:407] Connection header received
0925 10:02:42.021740 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.021741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.021754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.021813 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.022248 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.022747 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.023495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.023715 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.023792 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.025183 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.025184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.025184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.025191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.027158 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.027566 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.027567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.027568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.027654 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.027900 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.027901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.027902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.028099 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.028208 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.028622 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.028751 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:42.053853 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46317
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:42.141968 20967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:42.142254 20967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:42.142298 20967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:42.143623 20967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:42.143710 20967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:42.143749 20967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:42.143793 20967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:42.145509 20967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:42.145840 20967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:42.146173 20967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:42.151889 20975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:42.151993 20973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:42.151993 20972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:42.152376 20967 server_base.cc:1034] running on GCE node
I20260925 10:02:42.152585 20967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:42.152850 20967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:42.154054 20967 hybrid_clock.cc:648] HybridClock initialized: now 1790330562154031 us; error 41 us; skew 500 ppm
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:42.157189 20967 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:42.157796 20967 webserver.cc:460] Webserver started at http://127.19.46.194:32779/ using document root <none> and password file <none>
I20260925 10:02:42.158062 20967 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:42.158111 20967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:42.158255 20967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:42.159467 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "9bc1e41a92a9459a8964e8c45746ae09"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "1295cdad13160f1447061ab6b3796879"
server_key_iv: "b34c9a9bbc8cb3c90fb04b2533185671"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.159885 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "9bc1e41a92a9459a8964e8c45746ae09"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "1295cdad13160f1447061ab6b3796879"
server_key_iv: "b34c9a9bbc8cb3c90fb04b2533185671"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.161403 20967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:42.162554 20982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.162827 20967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:42.162964 20967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "9bc1e41a92a9459a8964e8c45746ae09"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "1295cdad13160f1447061ab6b3796879"
server_key_iv: "b34c9a9bbc8cb3c90fb04b2533185671"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.163051 20967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:42.173096 20967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:42.176435 20967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:42.176743 20967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:42.177340 20967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:42.177413 20967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.177528 20967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:42.177573 20967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.185117 20967 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37121
I20260925 10:02:42.185225 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37121 every 8 connection(s)
I20260925 10:02:42.185555 20967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:42.189728 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.178054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:58961 (local address 127.19.46.254:46317)
0925 10:02:42.178165 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.178169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.178288 (+ 119us) server_negotiation.cc:407] Connection header received
0925 10:02:42.178853 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.178854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.178869 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.178961 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.179925 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.181133 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.183025 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.183372 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.183497 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.186664 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.186667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.186667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.186677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.188044 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.188518 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.188519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.188520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.188599 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.188889 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.188890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.188890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.189032 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.189119 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.189497 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.189627 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:02:42.189905 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.177984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.194:58961)
0925 10:02:42.178158 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.178202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.178739 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.179033 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.179037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.179456 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.179784 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.179792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.181271 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.181272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.182813 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.182816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.183510 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.183512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.183703 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.184467 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.184479 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.186533 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.188158 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.188165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.188167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.188415 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.188673 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.188674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.188675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.188805 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.189106 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.189114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.189294 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.189424 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.189638 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260925 10:02:42.189952 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 20967
I20260925 10:02:42.190018 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:42.190209 19643 external_mini_cluster.cc:1428] Setting key 38bfe787393c253e6d2c309c99534253
I20260925 10:02:42.190495 21098 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46317
I20260925 10:02:42.190591 21098 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:42.190805 21098 heartbeater.cc:511] Master 127.19.46.254:46317 requested a full tablet report, sending...
I20260925 10:02:42.191273 20755 ts_manager.cc:194] Registered new tserver with Master: 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121)
I20260925 10:02:42.191869 20755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:58961
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:02:42.199640 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.192639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:43559 (local address 127.19.46.254:46317)
0925 10:02:42.192713 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.192716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.192727 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:42.192751 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.192751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.192762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.192833 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.193235 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.193869 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.194713 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.194953 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.195040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.196240 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.196242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.196242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.196250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.198236 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.198656 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.198658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.198658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.198723 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.198936 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.198937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.198937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.199145 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.199249 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.199421 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.199519 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:02:42.199637 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.191749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.194:43559)
0925 10:02:42.191975 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.191989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.192046 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.192855 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.192857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.193076 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.193142 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.193144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.194007 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.194009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.194602 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.194604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.195037 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.195038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.195142 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.195728 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.195733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.196120 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.198362 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.198363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.198364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.198560 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.198793 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.198794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.198795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.198861 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.199238 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.199240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.199305 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.199393 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.199514 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:42.226164 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46317
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:42.322408 21107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:42.322662 21107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:42.322693 21107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:42.324123 21107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:42.324182 21107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:42.324208 21107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:42.324226 21107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:42.325832 21107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:42.326104 21107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:42.326386 21107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:42.332100 21115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:42.332140 21113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:42.332067 21112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:42.332358 21107 server_base.cc:1034] running on GCE node
I20260925 10:02:42.332624 21107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:42.332897 21107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:42.334092 21107 hybrid_clock.cc:648] HybridClock initialized: now 1790330562334072 us; error 42 us; skew 500 ppm
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:42.336843 21107 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:42.337361 21107 webserver.cc:460] Webserver started at http://127.19.46.195:34183/ using document root <none> and password file <none>
I20260925 10:02:42.337694 21107 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:42.337771 21107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:42.337937 21107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:42.339172 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "d233acac58ad419787a49be4f271ce86"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "31d896ca026dba5598759eade7b9a31a"
server_key_iv: "b5d2a26a3c4cf9941e3e464bee8bda0b"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.339576 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "d233acac58ad419787a49be4f271ce86"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "31d896ca026dba5598759eade7b9a31a"
server_key_iv: "b5d2a26a3c4cf9941e3e464bee8bda0b"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.341409 21107 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:02:42.342553 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.342882 21107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:42.343112 21107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "d233acac58ad419787a49be4f271ce86"
format_stamp: "Formatted at 2026-09-25 10:02:42 on dist-test-slave-f9rl"
server_key: "31d896ca026dba5598759eade7b9a31a"
server_key_iv: "b5d2a26a3c4cf9941e3e464bee8bda0b"
server_key_version: "encryptionkey@0"
I20260925 10:02:42.343246 21107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:42.357926 21107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:42.361155 21107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:42.361368 21107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:42.362494 21107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:42.362617 21107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.362672 21107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:42.362707 21107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:42.372677 21107 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:44771
I20260925 10:02:42.372834 21237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:44771 every 8 connection(s)
I20260925 10:02:42.373212 21107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:42.374012 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 21107
I20260925 10:02:42.374114 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:42.374290 19643 external_mini_cluster.cc:1428] Setting key 1bf2bce02847907fb25fb487cd938930
I20260925 10:02:42.375952 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.363441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:38535 (local address 127.19.46.254:46317)
0925 10:02:42.363526 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.363529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.363713 (+ 184us) server_negotiation.cc:407] Connection header received
0925 10:02:42.364374 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.364376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.364393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.364531 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.365617 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.366626 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.368089 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.368323 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.368453 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.372003 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.372005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.372006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.372017 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.373951 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.374698 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.374700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.374701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.374774 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.375246 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.375247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.375248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.375422 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.375481 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.375792 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.375881 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:02:42.376107 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.363320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.195:38535)
0925 10:02:42.363545 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.363591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.364228 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.364645 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.364650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.365095 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.365429 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.365435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.366784 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.366786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.367950 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.367953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.368468 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.368469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.368623 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.369255 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.369267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.371838 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.374116 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.374124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.374126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.374583 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.374915 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.374916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.374916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.375118 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.375484 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.375490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.375630 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.375729 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.375871 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":141,"threads_started":1}
I20260925 10:02:42.376693 21238 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46317
I20260925 10:02:42.376788 21238 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:42.376997 21238 heartbeater.cc:511] Master 127.19.46.254:46317 requested a full tablet report, sending...
I20260925 10:02:42.377497 20754 ts_manager.cc:194] Registered new tserver with Master: d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771)
I20260925 10:02:42.378052 20754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:38535
I20260925 10:02:42.383978 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.378002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:59981 (local address 127.19.46.254:46317)
0925 10:02:42.378051 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.378053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.378124 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:02:42.378294 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.378295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.378308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.378377 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.378829 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.379310 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.380008 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.380214 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.380272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.381386 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.381387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.381388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.381394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.382952 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.383354 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.383355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.383355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.383410 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.383577 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.383577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.383577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.383682 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.383731 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.383832 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.383910 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:02:42.383996 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.377927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.19.46.195:59981)
0925 10:02:42.378065 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.378078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.378141 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.378393 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.378413 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.378650 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.378728 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.378731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.379409 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.379410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.379913 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.379915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.380300 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.380301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.380375 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.380913 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.380920 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.381287 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.383078 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.383079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.383079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.383275 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.383471 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.383472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.383472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.383526 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.383728 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.383728 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.383768 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.383823 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.383914 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260925 10:02:42.386276 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:42.413499 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.403589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.19.46.254:46317)
0925 10:02:42.403660 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.403662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.403671 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:02:42.403718 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.403719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.403729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.403792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.404800 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.405364 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.406319 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.406563 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.406653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.409633 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.409635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.409635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.409646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.411544 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.412123 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.412124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.412125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.412215 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.412530 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.412532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.412532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.412707 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.412811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.413245 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.413342 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:02:42.416124 20754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:42.417940 20754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:42.427490 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.421940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.19.46.194:37121)
0925 10:02:42.422186 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.422193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.422333 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:02:42.422882 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.422885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.422907 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.422983 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.423190 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424188 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.426142 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.426869 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.426949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427016 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.427303 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.427357 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260925 10:02:42.427484 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.421767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37121 (local address 127.0.0.1:34370)
0925 10:02:42.422242 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.422254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.422756 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.423061 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.423062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.423064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.423107 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.423108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424387 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.424388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.425993 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.425997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.427056 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.427058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.427163 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427180 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.427268 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.427335 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":58,"thread_start_us":140,"threads_started":1}
I20260925 10:02:42.427867 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.421723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:44771 (local address 127.0.0.1:54320)
0925 10:02:42.422044 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.422099 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.422196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.422704 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.422708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.422711 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.422844 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.422847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424322 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.424323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.426458 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.426461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.427579 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.427581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.427654 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427659 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.427743 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.427799 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":949,"thread_start_us":146,"threads_started":1}
I20260925 10:02:42.427883 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.422259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.19.46.193:43045)
0925 10:02:42.422426 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.422432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.422554 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:02:42.422659 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.422662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.422681 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.422777 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.422958 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424186 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.426109 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.426958 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.427032 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427099 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.427530 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.427774 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260925 10:02:42.428237 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.421826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.19.46.195:44771)
0925 10:02:42.422467 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.422472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.422488 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:02:42.422540 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.422543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.422565 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.422657 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.422941 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424193 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.426581 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.427399 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.427483 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427539 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.428055 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.428107 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":100,"threads_started":1}
I20260925 10:02:42.428499 21032 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce858a15a04baf974d21203618c743 (DEFAULT_TABLE table=test-table [id=41372a2e391f4ce89213d8ac373a5197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.428992 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce858a15a04baf974d21203618c743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.428936 21172 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce858a15a04baf974d21203618c743 (DEFAULT_TABLE table=test-table [id=41372a2e391f4ce89213d8ac373a5197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.429361 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce858a15a04baf974d21203618c743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.432575 21261 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Bootstrap starting.
I20260925 10:02:42.432665 21262 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Bootstrap starting.
I20260925 10:02:42.427484 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.422065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43045 (local address 127.0.0.1:58618)
0925 10:02:42.422453 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.422465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.422572 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.422774 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.422775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.422778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.422870 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.422872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.424593 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.424594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.425993 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.425997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.427136 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.427137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.427231 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.427235 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.427313 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.427358 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":6,"thread_start_us":89,"threads_started":1}
I20260925 10:02:42.433514 21262 tablet_bootstrap.cc:654] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.433868 21262 log.cc:824] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:42.433828 20884 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce858a15a04baf974d21203618c743 (DEFAULT_TABLE table=test-table [id=41372a2e391f4ce89213d8ac373a5197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.434094 21261 tablet_bootstrap.cc:654] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.434263 20884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce858a15a04baf974d21203618c743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.434464 21261 log.cc:824] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:42.434788 21262 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: No bootstrap required, opened a new log
I20260925 10:02:42.434875 21262 ts_tablet_manager.cc:1397] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:02:42.435243 21261 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: No bootstrap required, opened a new log
I20260925 10:02:42.435314 21261 ts_tablet_manager.cc:1397] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:02:42.437062 21262 raft_consensus.cc:348] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.437253 21262 raft_consensus.cc:374] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.437289 21262 raft_consensus.cc:729] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc1e41a92a9459a8964e8c45746ae09, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.437498 21262 consensus_queue.cc:260] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.437424 21261 raft_consensus.cc:348] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.437623 21261 raft_consensus.cc:374] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.437667 21261 raft_consensus.cc:729] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d233acac58ad419787a49be4f271ce86, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.437781 21261 consensus_queue.cc:260] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.437835 21262 ts_tablet_manager.cc:1428] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:02:42.438032 21261 ts_tablet_manager.cc:1428] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:02:42.438055 21098 heartbeater.cc:503] Master 127.19.46.254:46317 was elected leader, sending a full tablet report...
I20260925 10:02:42.438158 21238 heartbeater.cc:503] Master 127.19.46.254:46317 was elected leader, sending a full tablet report...
I20260925 10:02:42.438099 21265 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Bootstrap starting.
I20260925 10:02:42.439128 21265 tablet_bootstrap.cc:654] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.439668 21265 log.cc:824] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:42.440800 21265 tablet_bootstrap.cc:492] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: No bootstrap required, opened a new log
I20260925 10:02:42.440879 21265 ts_tablet_manager.cc:1397] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:02:42.443215 21265 raft_consensus.cc:348] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.443444 21265 raft_consensus.cc:374] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.443491 21265 raft_consensus.cc:729] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390a90a13ed542bf8cc91b1c1a121ac2, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.443658 21265 consensus_queue.cc:260] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.443984 21265 ts_tablet_manager.cc:1428] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:02:42.444090 20958 heartbeater.cc:503] Master 127.19.46.254:46317 was elected leader, sending a full tablet report...
W20260925 10:02:42.515203 20959 tablet.cc:2369] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:42.618446 21269 raft_consensus.cc:482] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:42.618639 21269 raft_consensus.cc:504] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.619144 21269 leader_election.cc:288] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771), 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121)
I20260925 10:02:42.622242 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.619466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:57885 (local address 127.19.46.194:37121)
0925 10:02:42.619516 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.619518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.619621 (+ 103us) server_negotiation.cc:407] Connection header received
0925 10:02:42.619700 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.619702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.619711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.619766 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.620007 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.620456 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.621487 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.621879 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.621937 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.621945 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.622120 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.622168 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:02:42.622262 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.619376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37121 (local address 127.19.46.193:57885)
0925 10:02:42.619531 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.619542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.619613 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.619779 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.619780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.619784 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.619898 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.619900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.620566 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.620567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.621383 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.621386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.621961 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.621962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.622039 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.622044 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.622096 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.622147 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":18,"thread_start_us":103,"threads_started":1}
I20260925 10:02:42.622330 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.619468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:48509 (local address 127.19.46.195:44771)
0925 10:02:42.619516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.619518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.619656 (+ 138us) server_negotiation.cc:407] Connection header received
0925 10:02:42.619696 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.619697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.619708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.619786 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.619948 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.620443 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.621560 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.621950 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.622023 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.622032 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.622213 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.622254 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:02:42.622571 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.619380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:44771 (local address 127.19.46.193:48509)
0925 10:02:42.619540 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.619550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.619642 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.619796 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.619797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.619799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.619878 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.619880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.620534 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.620536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.621395 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.621397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.622041 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.622042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.622128 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.622132 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.622188 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.622230 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":38,"thread_start_us":69,"threads_started":1}
I20260925 10:02:42.622629 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce858a15a04baf974d21203618c743" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09" is_pre_election: true
I20260925 10:02:42.622810 21052 raft_consensus.cc:2454] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 0.
I20260925 10:02:42.622884 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce858a15a04baf974d21203618c743" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233acac58ad419787a49be4f271ce86" is_pre_election: true
I20260925 10:02:42.623056 21192 raft_consensus.cc:2454] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 0.
I20260925 10:02:42.623080 20844 leader_election.cc:302] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390a90a13ed542bf8cc91b1c1a121ac2, 9bc1e41a92a9459a8964e8c45746ae09; no voters:
I20260925 10:02:42.623219 21269 raft_consensus.cc:2781] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:42.623292 21269 raft_consensus.cc:482] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:42.623322 21269 raft_consensus.cc:3037] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:02:42.623785 21239 tablet.cc:2369] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:42.624109 21269 raft_consensus.cc:504] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.624265 21269 leader_election.cc:288] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 election: Requested vote from peers d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771), 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121)
I20260925 10:02:42.624471 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce858a15a04baf974d21203618c743" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233acac58ad419787a49be4f271ce86"
I20260925 10:02:42.624459 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce858a15a04baf974d21203618c743" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09"
I20260925 10:02:42.624544 21192 raft_consensus.cc:3037] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.624547 21052 raft_consensus.cc:3037] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.625424 21052 raft_consensus.cc:2454] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 1.
I20260925 10:02:42.625424 21192 raft_consensus.cc:2454] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 1.
I20260925 10:02:42.625670 20846 leader_election.cc:302] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390a90a13ed542bf8cc91b1c1a121ac2, d233acac58ad419787a49be4f271ce86; no voters:
I20260925 10:02:42.625777 21269 raft_consensus.cc:2781] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:42.625900 21269 raft_consensus.cc:686] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 390a90a13ed542bf8cc91b1c1a121ac2, State: Running, Role: LEADER
I20260925 10:02:42.626024 21269 consensus_queue.cc:237] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } }
I20260925 10:02:42.626917 20755 catalog_manager.cc:5495] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260925 10:02:42.657977 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.655183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.19.46.193:43045)
0925 10:02:42.655253 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.655257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.655270 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:02:42.655308 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.655309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.655327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.655388 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:42.655659 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.656150 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.657020 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.657269 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.657340 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.657594 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.657703 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.657770 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.657886 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.657909 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:02:42.660017 21192 raft_consensus.cc:1261] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Refusing update from remote peer 390a90a13ed542bf8cc91b1c1a121ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.660022 21052 raft_consensus.cc:1261] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Refusing update from remote peer 390a90a13ed542bf8cc91b1c1a121ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.660441 21272 consensus_queue.cc:1035] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.660600 21269 consensus_queue.cc:1035] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.664322 21277 mvcc.cc:205] Tried to move back new op lower bound from 7333193984652365824 to 7333193984516849664. Current Snapshot: MvccSnapshot[applied={T|T < 7333193984652365824 or (T in {7333193984652365824})}]
I20260925 10:02:42.667517 20754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:42.667649 20754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:42.668550 20754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Sending DeleteTablet for 3 replicas of tablet a3ce858a15a04baf974d21203618c743
I20260925 10:02:42.668967 21032 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce858a15a04baf974d21203618c743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:34370
I20260925 10:02:42.669005 21172 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce858a15a04baf974d21203618c743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:54320
I20260925 10:02:42.669243 21286 tablet_replica.cc:330] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: stopping tablet replica
I20260925 10:02:42.669283 20884 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce858a15a04baf974d21203618c743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:58618
I20260925 10:02:42.669373 21286 raft_consensus.cc:2229] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:42.669529 21286 raft_consensus.cc:2258] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.669517 20754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:42.669711 20754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:42.669744 21285 tablet_replica.cc:330] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: stopping tablet replica
I20260925 10:02:42.669844 21285 raft_consensus.cc:2229] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:42.669958 21287 tablet_replica.cc:330] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: stopping tablet replica
I20260925 10:02:42.670066 21287 raft_consensus.cc:2229] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:02:42.670115 21285 raft_consensus.cc:2258] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.670184 21287 raft_consensus.cc:2258] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.670667 21285 ts_tablet_manager.cc:1905] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.671322 21286 ts_tablet_manager.cc:1905] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.671828 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (DEFAULT_TABLE table=test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.671954 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.673024 21285 ts_tablet_manager.cc:1918] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:02:42.673096 21285 log.cc:1196] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/a3ce858a15a04baf974d21203618c743
I20260925 10:02:42.673303 21286 ts_tablet_manager.cc:1918] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:02:42.673372 21286 log.cc:1196] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/a3ce858a15a04baf974d21203618c743
I20260925 10:02:42.673401 21285 ts_tablet_manager.cc:1939] T a3ce858a15a04baf974d21203618c743 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting consensus metadata
I20260925 10:02:42.673514 21261 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Bootstrap starting.
I20260925 10:02:42.673679 21286 ts_tablet_manager.cc:1939] T a3ce858a15a04baf974d21203618c743 P d233acac58ad419787a49be4f271ce86: Deleting consensus metadata
I20260925 10:02:42.673985 21261 tablet_bootstrap.cc:654] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.674436 20742 catalog_manager.cc:4841] TS d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771): tablet a3ce858a15a04baf974d21203618c743 (table test-table [id=41372a2e391f4ce89213d8ac373a5197]) successfully deleted
I20260925 10:02:42.674612 21261 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: No bootstrap required, opened a new log
I20260925 10:02:42.674661 21261 ts_tablet_manager.cc:1397] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:42.674739 20740 catalog_manager.cc:4841] TS 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121): tablet a3ce858a15a04baf974d21203618c743 (table test-table [id=41372a2e391f4ce89213d8ac373a5197]) successfully deleted
I20260925 10:02:42.674799 21261 raft_consensus.cc:348] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.674870 21261 raft_consensus.cc:374] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.674891 21261 raft_consensus.cc:729] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d233acac58ad419787a49be4f271ce86, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.674938 21261 consensus_queue.cc:260] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.674945 20884 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (DEFAULT_TABLE table=test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.675021 21261 ts_tablet_manager.cc:1428] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.675029 21032 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (DEFAULT_TABLE table=test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:42.675071 20884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.675123 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.676465 21265 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Bootstrap starting.
I20260925 10:02:42.676491 21262 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Bootstrap starting.
I20260925 10:02:42.676935 21265 tablet_bootstrap.cc:654] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.676964 21262 tablet_bootstrap.cc:654] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.677713 21262 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: No bootstrap required, opened a new log
I20260925 10:02:42.677765 21262 ts_tablet_manager.cc:1397] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:42.677938 21262 raft_consensus.cc:348] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.678027 21262 raft_consensus.cc:374] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.678054 21262 raft_consensus.cc:729] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc1e41a92a9459a8964e8c45746ae09, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.678112 21262 consensus_queue.cc:260] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.678231 21262 ts_tablet_manager.cc:1428] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.678437 21287 ts_tablet_manager.cc:1905] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.680042 21265 tablet_bootstrap.cc:492] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: No bootstrap required, opened a new log
I20260925 10:02:42.680105 21265 ts_tablet_manager.cc:1397] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 10:02:42.680264 21265 raft_consensus.cc:348] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.680290 21287 ts_tablet_manager.cc:1918] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:02:42.680363 21265 raft_consensus.cc:374] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.680385 21265 raft_consensus.cc:729] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390a90a13ed542bf8cc91b1c1a121ac2, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.680410 21287 log.cc:1196] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/a3ce858a15a04baf974d21203618c743
I20260925 10:02:42.680434 21265 consensus_queue.cc:260] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.680521 21265 ts_tablet_manager.cc:1428] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:42.680727 21287 ts_tablet_manager.cc:1939] T a3ce858a15a04baf974d21203618c743 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting consensus metadata
I20260925 10:02:42.681154 20740 catalog_manager.cc:4841] TS 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193:43045): tablet a3ce858a15a04baf974d21203618c743 (table test-table [id=41372a2e391f4ce89213d8ac373a5197]) successfully deleted
W20260925 10:02:42.686421 21099 tablet.cc:2369] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:42.745414 21267 raft_consensus.cc:482] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:42.745560 21267 raft_consensus.cc:504] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.745922 21267 leader_election.cc:288] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193:43045), 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121)
I20260925 10:02:42.749694 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.746267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:60713 (local address 127.19.46.193:43045)
0925 10:02:42.746309 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.746311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.746322 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:42.746436 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.746437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.746448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.746499 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.746711 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.747331 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.748777 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.749325 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.749402 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.749411 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.749577 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.749617 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:02:42.749711 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.746174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43045 (local address 127.19.46.195:60713)
0925 10:02:42.746241 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.746255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.746325 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.746552 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.746553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.746554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.746650 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.746652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.747441 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.747442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.748685 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.748687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.749420 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.749421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.749508 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.749510 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.749562 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.749636 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":2}
I20260925 10:02:42.749719 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.746456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:48937 (local address 127.19.46.194:37121)
0925 10:02:42.746499 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.746501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.746509 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:42.746537 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.746538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.746547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.746594 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:42.746841 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.747563 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.748909 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.749281 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.749366 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.749376 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.749611 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.749660 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:02:42.750087 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.746279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37121 (local address 127.19.46.195:48937)
0925 10:02:42.746412 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.746422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.746486 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.746629 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.746630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.746632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:42.746742 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.746744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.747655 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.747656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.748828 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.748830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.749400 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.749401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.749501 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.749505 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.749562 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.750016 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":417,"thread_start_us":63,"threads_started":1}
I20260925 10:02:42.750244 20912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf4d39d7eb44efba7e1f0b3f3c058a2" candidate_uuid: "d233acac58ad419787a49be4f271ce86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" is_pre_election: true
I20260925 10:02:42.750384 20912 raft_consensus.cc:2454] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d233acac58ad419787a49be4f271ce86 in term 0.
I20260925 10:02:42.750594 21124 leader_election.cc:302] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390a90a13ed542bf8cc91b1c1a121ac2, d233acac58ad419787a49be4f271ce86; no voters:
I20260925 10:02:42.750658 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf4d39d7eb44efba7e1f0b3f3c058a2" candidate_uuid: "d233acac58ad419787a49be4f271ce86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09" is_pre_election: true
I20260925 10:02:42.750738 21267 raft_consensus.cc:2781] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:42.750751 21052 raft_consensus.cc:2454] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d233acac58ad419787a49be4f271ce86 in term 0.
I20260925 10:02:42.750777 21267 raft_consensus.cc:482] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:42.750799 21267 raft_consensus.cc:3037] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.751513 21267 raft_consensus.cc:504] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.751673 21267 leader_election.cc:288] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [CANDIDATE]: Term 1 election: Requested vote from peers 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193:43045), 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121)
I20260925 10:02:42.751829 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf4d39d7eb44efba7e1f0b3f3c058a2" candidate_uuid: "d233acac58ad419787a49be4f271ce86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09"
I20260925 10:02:42.751858 20912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf4d39d7eb44efba7e1f0b3f3c058a2" candidate_uuid: "d233acac58ad419787a49be4f271ce86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390a90a13ed542bf8cc91b1c1a121ac2"
I20260925 10:02:42.751925 21052 raft_consensus.cc:3037] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.751940 20912 raft_consensus.cc:3037] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.752516 21052 raft_consensus.cc:2454] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d233acac58ad419787a49be4f271ce86 in term 1.
I20260925 10:02:42.752679 21124 leader_election.cc:302] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc1e41a92a9459a8964e8c45746ae09, d233acac58ad419787a49be4f271ce86; no voters:
I20260925 10:02:42.752766 21267 raft_consensus.cc:2781] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:42.752799 20912 raft_consensus.cc:2454] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d233acac58ad419787a49be4f271ce86 in term 1.
I20260925 10:02:42.752919 21267 raft_consensus.cc:686] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 1 LEADER]: Becoming Leader. State: Replica: d233acac58ad419787a49be4f271ce86, State: Running, Role: LEADER
I20260925 10:02:42.753000 21267 consensus_queue.cc:237] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.753799 20755 catalog_manager.cc:5495] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 reported cstate change: term changed from 0 to 1, leader changed from <none> to d233acac58ad419787a49be4f271ce86 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "d233acac58ad419787a49be4f271ce86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } health_report { overall_health: HEALTHY } } }
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330561, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:42.790294 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.782344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.19.46.254:46317)
0925 10:02:42.782413 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.782415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.782512 (+ 97us) server_negotiation.cc:407] Connection header received
0925 10:02:42.782586 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.782587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.782599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.782658 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.783121 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.783846 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.784614 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.784926 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.785006 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.787329 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.787331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.787331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.787339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.788926 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.789372 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.789373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.789373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.789461 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.789691 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.789692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.789693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.789834 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.789905 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.790052 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.790163 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:02:42.790390 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.782260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46317 (local address 127.0.0.1:43502)
0925 10:02:42.782444 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.782453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.782511 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.782676 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.782677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.782972 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:42.783030 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.783032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.783976 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.783995 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.784527 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.784528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.785042 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.785042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.785091 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.785770 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:42.785775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:42.787236 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:42.789030 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.789038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.789040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.789283 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.789569 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:42.789570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:42.789570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:42.789627 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:42.789895 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:42.789901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:42.789946 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.790027 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.790235 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":73,"thread_start_us":101,"threads_started":1}
I20260925 10:02:42.792906 20755 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:43502:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:02:42.793116 20755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:42.794628 20884 tablet_service.cc:1467] Processing CreateTablet for tablet 5c793ac9809d4882b1930484c519de0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46ef322227524e898537232b329b4ca8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:42.794668 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 5c793ac9809d4882b1930484c519de0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46ef322227524e898537232b329b4ca8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:42.794806 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c793ac9809d4882b1930484c519de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.794806 20884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c793ac9809d4882b1930484c519de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.795794 21032 tablet_service.cc:1467] Processing CreateTablet for tablet 5c793ac9809d4882b1930484c519de0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46ef322227524e898537232b329b4ca8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:42.795955 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c793ac9809d4882b1930484c519de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:42.796298 21261 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: Bootstrap starting.
I20260925 10:02:42.796756 21261 tablet_bootstrap.cc:654] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.797180 21265 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: Bootstrap starting.
I20260925 10:02:42.797515 21265 tablet_bootstrap.cc:654] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.797609 21261 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: No bootstrap required, opened a new log
I20260925 10:02:42.797662 21262 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: Bootstrap starting.
I20260925 10:02:42.797669 21261 ts_tablet_manager.cc:1397] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:42.797828 21261 raft_consensus.cc:348] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.797905 21261 raft_consensus.cc:374] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.797930 21261 raft_consensus.cc:729] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d233acac58ad419787a49be4f271ce86, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.797973 21261 consensus_queue.cc:260] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.798066 21261 ts_tablet_manager.cc:1428] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:02:42.798077 21262 tablet_bootstrap.cc:654] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:42.798197 21267 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.798230 21265 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: No bootstrap required, opened a new log
I20260925 10:02:42.798269 21267 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.798277 21265 ts_tablet_manager.cc:1397] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:42.798383 21265 raft_consensus.cc:348] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.798480 21265 raft_consensus.cc:374] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.798513 21265 raft_consensus.cc:729] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390a90a13ed542bf8cc91b1c1a121ac2, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.798566 21265 consensus_queue.cc:260] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.798645 21262 tablet_bootstrap.cc:492] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: No bootstrap required, opened a new log
I20260925 10:02:42.798671 21265 ts_tablet_manager.cc:1428] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.798681 21262 ts_tablet_manager.cc:1397] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:42.798708 21282 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.798820 21282 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.798817 21262 raft_consensus.cc:348] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.798911 21262 raft_consensus.cc:374] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:42.798942 21262 raft_consensus.cc:729] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc1e41a92a9459a8964e8c45746ae09, State: Initialized, Role: FOLLOWER
I20260925 10:02:42.799000 21262 consensus_queue.cc:260] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.799111 21262 ts_tablet_manager.cc:1428] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:42.799170 21266 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.799258 21266 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.890096 21282 raft_consensus.cc:482] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:42.890202 21282 raft_consensus.cc:504] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.890432 21282 leader_election.cc:288] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121), d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771)
I20260925 10:02:42.890689 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c793ac9809d4882b1930484c519de0f" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09" is_pre_election: true
I20260925 10:02:42.890798 21052 raft_consensus.cc:2454] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 0.
I20260925 10:02:42.890726 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c793ac9809d4882b1930484c519de0f" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233acac58ad419787a49be4f271ce86" is_pre_election: true
I20260925 10:02:42.890872 21192 raft_consensus.cc:2454] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 0.
I20260925 10:02:42.890951 20844 leader_election.cc:302] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390a90a13ed542bf8cc91b1c1a121ac2, 9bc1e41a92a9459a8964e8c45746ae09; no voters:
I20260925 10:02:42.891055 21282 raft_consensus.cc:2781] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:42.891105 21282 raft_consensus.cc:482] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:42.891135 21282 raft_consensus.cc:3037] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.891959 21282 raft_consensus.cc:504] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.892112 21282 leader_election.cc:288] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121), d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771)
I20260925 10:02:42.892259 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c793ac9809d4882b1930484c519de0f" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc1e41a92a9459a8964e8c45746ae09"
I20260925 10:02:42.892315 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c793ac9809d4882b1930484c519de0f" candidate_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233acac58ad419787a49be4f271ce86"
I20260925 10:02:42.892369 21052 raft_consensus.cc:3037] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.892371 21192 raft_consensus.cc:3037] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:42.893013 21192 raft_consensus.cc:2454] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 1.
I20260925 10:02:42.893039 21052 raft_consensus.cc:2454] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390a90a13ed542bf8cc91b1c1a121ac2 in term 1.
I20260925 10:02:42.893190 20846 leader_election.cc:302] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390a90a13ed542bf8cc91b1c1a121ac2, d233acac58ad419787a49be4f271ce86; no voters:
I20260925 10:02:42.893292 21282 raft_consensus.cc:2781] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:42.893360 21282 raft_consensus.cc:686] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 390a90a13ed542bf8cc91b1c1a121ac2, State: Running, Role: LEADER
I20260925 10:02:42.893421 21282 consensus_queue.cc:237] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } }
I20260925 10:02:42.893791 21269 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: TxnStatusTablet state changed. Reason: New leader 390a90a13ed542bf8cc91b1c1a121ac2. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.893890 21269 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:42.894116 21304 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:02:42.894140 20754 catalog_manager.cc:5495] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } health_report { overall_health: UNKNOWN } } }
I20260925 10:02:42.894248 21304 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:02:42.917167 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.914365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43045 (local address 127.0.0.1:58634)
0925 10:02:42.914447 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.914460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.914531 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.914672 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.914673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.914674 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:42.914751 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.914753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.915674 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.915674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.916249 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.916250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.916730 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.916731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.916791 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.916809 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.917000 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.917007 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.917043 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.917094 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:02:42.917163 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.914457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.19.46.193:43045)
0925 10:02:42.914509 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.914511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.914522 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:42.914592 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.914593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.914604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.914668 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:42.914818 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.915586 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.916331 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.916636 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.916705 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.916901 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.916937 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.916990 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.917055 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.917076 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:02:42.918362 21192 raft_consensus.cc:1261] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Refusing update from remote peer 390a90a13ed542bf8cc91b1c1a121ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.918434 21052 raft_consensus.cc:1261] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Refusing update from remote peer 390a90a13ed542bf8cc91b1c1a121ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.918718 21282 consensus_queue.cc:1035] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.918877 21269 consensus_queue.cc:1035] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.919200 21267 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: TxnStatusTablet state changed. Reason: New leader 390a90a13ed542bf8cc91b1c1a121ac2. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.919304 21267 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.919418 21266 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: TxnStatusTablet state changed. Reason: New leader 390a90a13ed542bf8cc91b1c1a121ac2. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.919514 21266 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.920022 21275 mvcc.cc:205] Tried to move back new op lower bound from 7333193985711497216 to 7333193985611907072. Current Snapshot: MvccSnapshot[applied={T|T < 7333193985711497216}]
I20260925 10:02:42.920128 21272 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.920209 21272 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:42.920255 21272 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.920295 21272 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 390a90a13ed542bf8cc91b1c1a121ac2: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:42.920362 21276 mvcc.cc:205] Tried to move back new op lower bound from 7333193985711497216 to 7333193985611907072. Current Snapshot: MvccSnapshot[applied={T|T < 7333193985711497216 or (T in {7333193985711497216})}]
I20260925 10:02:42.920482 21267 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.920573 21267 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P d233acac58ad419787a49be4f271ce86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.920617 21266 tablet_replica.cc:439] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 } } peers { permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 } } peers { permanent_uuid: "d233acac58ad419787a49be4f271ce86" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 44771 } } }
I20260925 10:02:42.920689 21266 tablet_replica.cc:442] T 5c793ac9809d4882b1930484c519de0f P 9bc1e41a92a9459a8964e8c45746ae09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:42.924928 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.922284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.19.46.195:44771)
0925 10:02:42.922330 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.922332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.922358 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:02:42.922454 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.922455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.922474 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.922580 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:42.922733 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.923259 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.924049 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.924285 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.924367 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.924540 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.924669 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.924760 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.924807 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.924828 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:02:42.930619 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.927630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43045 (local address 127.19.46.195:41431)
0925 10:02:42.927876 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.927886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.927944 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.928068 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.928069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.928070 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:42.928125 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.928126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.928972 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.928973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.929693 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.929695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.930217 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.930218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.930277 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.930293 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.930476 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.930478 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.930514 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.930550 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260925 10:02:42.930619 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.927716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:41431 (local address 127.19.46.193:43045)
0925 10:02:42.927792 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.927794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.927953 (+ 159us) server_negotiation.cc:407] Connection header received
0925 10:02:42.927986 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.927986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.927998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.928067 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:42.928171 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.928889 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.929774 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.930078 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.930153 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.930382 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.930415 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.930476 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.930517 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.930539 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:02:42.933346 21052 raft_consensus.cc:1261] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Refusing update from remote peer d233acac58ad419787a49be4f271ce86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.933506 20912 raft_consensus.cc:1261] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Refusing update from remote peer d233acac58ad419787a49be4f271ce86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:42.933862 21292 consensus_queue.cc:1035] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc1e41a92a9459a8964e8c45746ae09" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.934141 21292 consensus_queue.cc:1035] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "390a90a13ed542bf8cc91b1c1a121ac2" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:42.943455 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.940745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:41887 (local address 127.19.46.195:44771)
0925 10:02:42.940788 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.940790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.940879 (+ 89us) server_negotiation.cc:407] Connection header received
0925 10:02:42.940950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.940950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.940958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.941002 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:42.941124 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.941717 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.942805 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.942987 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.943040 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.943235 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.943263 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.943300 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.943382 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.943400 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:02:42.943536 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.940646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:44771 (local address 127.19.46.193:41887)
0925 10:02:42.940818 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:42.940830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:42.940892 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:42.941007 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:42.941008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:42.941010 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:42.941068 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.941070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.941799 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.941800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.942719 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:42.942722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.943065 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.943066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:42.943138 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.943157 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:42.943324 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:42.943326 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:42.943380 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:02:42.943427 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 10:02:42.952903 20754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:42.953032 20754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:42.953727 20754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e380e4e2d79c46ec852e75a65641cc70: Sending DeleteTablet for 3 replicas of tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2
I20260925 10:02:42.954003 21172 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:54320
I20260925 10:02:42.954095 21286 tablet_replica.cc:330] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: stopping tablet replica
I20260925 10:02:42.954139 21286 raft_consensus.cc:2229] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:02:42.954227 21286 raft_consensus.cc:2258] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.954291 21032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:34370
I20260925 10:02:42.954401 21285 tablet_replica.cc:330] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: stopping tablet replica
I20260925 10:02:42.954471 21285 raft_consensus.cc:2229] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:42.954470 20884 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:58618
I20260925 10:02:42.954514 21285 raft_consensus.cc:2258] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.954545 21286 ts_tablet_manager.cc:1905] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.954563 21287 tablet_replica.cc:330] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: stopping tablet replica
I20260925 10:02:42.954627 21287 raft_consensus.cc:2229] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:42.954664 21287 raft_consensus.cc:2258] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:42.954825 21285 ts_tablet_manager.cc:1905] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.954994 21287 ts_tablet_manager.cc:1905] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:42.956350 21286 ts_tablet_manager.cc:1918] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:42.956404 21286 log.cc:1196] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/0bf4d39d7eb44efba7e1f0b3f3c058a2
I20260925 10:02:42.956542 21285 ts_tablet_manager.cc:1918] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:42.956590 21285 log.cc:1196] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/0bf4d39d7eb44efba7e1f0b3f3c058a2
I20260925 10:02:42.956612 21287 ts_tablet_manager.cc:1918] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:42.956637 21286 ts_tablet_manager.cc:1939] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P d233acac58ad419787a49be4f271ce86: Deleting consensus metadata
I20260925 10:02:42.956666 21287 log.cc:1196] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/0bf4d39d7eb44efba7e1f0b3f3c058a2
I20260925 10:02:42.956795 21285 ts_tablet_manager.cc:1939] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 9bc1e41a92a9459a8964e8c45746ae09: Deleting consensus metadata
I20260925 10:02:42.956952 21287 ts_tablet_manager.cc:1939] T 0bf4d39d7eb44efba7e1f0b3f3c058a2 P 390a90a13ed542bf8cc91b1c1a121ac2: Deleting consensus metadata
I20260925 10:02:42.957160 20740 catalog_manager.cc:4841] TS 9bc1e41a92a9459a8964e8c45746ae09 (127.19.46.194:37121): tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (table test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]) successfully deleted
I20260925 10:02:42.957160 20742 catalog_manager.cc:4841] TS d233acac58ad419787a49be4f271ce86 (127.19.46.195:44771): tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (table test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]) successfully deleted
I20260925 10:02:42.957283 20740 catalog_manager.cc:4841] TS 390a90a13ed542bf8cc91b1c1a121ac2 (127.19.46.193:43045): tablet 0bf4d39d7eb44efba7e1f0b3f3c058a2 (table test-table [id=06ebd8bb036a4e53af4e9b1d9226900b]) successfully deleted
Sep 25 10:02:42 dist-test-slave-f9rl krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:42.971856 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.958454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.19.46.193:43045)
0925 10:02:42.958500 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.958502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.961697 (+ 3195us) server_negotiation.cc:407] Connection header received
0925 10:02:42.961743 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.961745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.961759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.961836 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.962543 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.963298 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.964318 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.964642 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.964714 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.968381 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.968395 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.968396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.968404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.970443 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.970890 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.970893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.970894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.970961 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.971199 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.971200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.971200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.971379 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.971530 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.971664 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.971748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:02:42.972370 20935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58636
I20260925 10:02:42.985177 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:42.974753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.19.46.254:46317)
0925 10:02:42.974826 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:42.974828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:42.977150 (+ 2322us) server_negotiation.cc:407] Connection header received
0925 10:02:42.977221 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:42.977222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:42.977236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:42.977305 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:42.978058 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.978956 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.980011 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:42.980317 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:42.980384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:42.982026 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:42.982028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:42.982028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:42.982036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:42.983814 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.984226 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.984227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.984227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.984285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:42.984551 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:42.984552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:42.984553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:42.984730 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:42.984834 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:42.984993 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:02:42.985072 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 10:02:42.985726 20754 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43512
I20260925 10:02:42.986675 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20827
I20260925 10:02:42.994623 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20967
I20260925 10:02:43.000164 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 21107
I20260925 10:02:43.005342 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 20723
2026-09-25T10:02:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2595 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:43 dist-test-slave-f9rl krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:43 dist-test-slave-f9rl krb5kdc[21324](info): set up 2 sockets
Sep 25 10:02:43 dist-test-slave-f9rl krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260925 10:02:44.074627 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:44.218508 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46293
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:44.306732 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:44.306952 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:44.306975 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:44.308107 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:44.308144 21337 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:02:44.308157 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:44.308171 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:44.308182 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:44.308194 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:44.309715 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:44.309916 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:44.310102 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:44.314930 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.314939 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.315124 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.315155 21337 server_base.cc:1034] running on GCE node
I20260925 10:02:44.315407 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:44.315721 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:44.316901 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790330564316894 us; error 41 us; skew 500 ppm
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:44.319620 21337 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:44.320114 21337 webserver.cc:460] Webserver started at http://127.19.46.254:43847/ using document root <none> and password file <none>
I20260925 10:02:44.320365 21337 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:44.320437 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:44.320592 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:44.321784 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "6f51075866e64d88bb06bcd9b2c8c8ae"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "32f47787184f7d24dd3447d5dde33472"
server_key_iv: "bf461759ee297dc57966efbc221a454a"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.322148 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "6f51075866e64d88bb06bcd9b2c8c8ae"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "32f47787184f7d24dd3447d5dde33472"
server_key_iv: "bf461759ee297dc57966efbc221a454a"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.323534 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:44.324311 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.324508 21337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:44.324612 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "6f51075866e64d88bb06bcd9b2c8c8ae"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "32f47787184f7d24dd3447d5dde33472"
server_key_iv: "bf461759ee297dc57966efbc221a454a"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.324710 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:44.360700 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:44.362807 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:44.362989 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:44.367795 21337 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46293
I20260925 10:02:44.367854 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46293 every 8 connection(s)
I20260925 10:02:44.368189 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:44.368831 21415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:44.371244 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Bootstrap starting.
I20260925 10:02:44.371862 21415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:44.372159 21415 log.cc:824] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:44.372901 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: No bootstrap required, opened a new log
I20260925 10:02:44.375046 21415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } }
I20260925 10:02:44.375197 21415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:44.375288 21415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f51075866e64d88bb06bcd9b2c8c8ae, State: Initialized, Role: FOLLOWER
I20260925 10:02:44.375439 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 21337
I20260925 10:02:44.375540 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:44.375506 21415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } }
I20260925 10:02:44.375622 21415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:44.375666 21415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:44.375713 19643 external_mini_cluster.cc:1428] Setting key 18de5dad3265570ef71e6dfff7c91e58
I20260925 10:02:44.375720 21415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:44.376410 21415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } }
I20260925 10:02:44.376499 21415 leader_election.cc:302] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f51075866e64d88bb06bcd9b2c8c8ae; no voters:
I20260925 10:02:44.376703 21415 leader_election.cc:288] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:44.376734 21420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:44.376885 21420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [term 1 LEADER]: Becoming Leader. State: Replica: 6f51075866e64d88bb06bcd9b2c8c8ae, State: Running, Role: LEADER
I20260925 10:02:44.376989 21415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:44.377005 21420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } }
I20260925 10:02:44.377643 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f51075866e64d88bb06bcd9b2c8c8ae. Latest consensus state: current_term: 1 leader_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } } }
I20260925 10:02:44.377621 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f51075866e64d88bb06bcd9b2c8c8ae" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46293 } } }
I20260925 10:02:44.377856 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:44.377880 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:44.378446 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:44.379029 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:44.380632 21432 catalog_manager.cc:1347] Generated new cluster ID: c67df0ddf2e04623b60586acba97c3d2
I20260925 10:02:44.380690 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:44.388530 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.376225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.19.46.254:46293)
0925 10:02:44.376422 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.376433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.376467 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:02:44.376613 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.376629 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.376721 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.377508 (+ 787us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.377970 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.379710 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.380674 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.381264 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.381367 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.384565 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.384570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.384571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.384584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.386709 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.387185 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.387189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.387189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.387309 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.387540 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.387541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.387541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.387859 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.388020 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.388184 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.388322 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 10:02:44.392699 21432 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:44.393280 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:44.399210 21432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Generated new TSK 0
I20260925 10:02:44.399439 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:44.432870 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46293
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:44.516428 21441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:44.516736 21441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:44.516794 21441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:44.518366 21441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:44.518430 21441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:44.518464 21441 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:02:44.518491 21441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:44.518522 21441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:44.520172 21441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:44.520437 21441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:44.520651 21441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:44.525748 21446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.525745 21449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.525871 21441 server_base.cc:1034] running on GCE node
W20260925 10:02:44.525777 21447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.526257 21441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:44.526458 21441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:44.527609 21441 hybrid_clock.cc:648] HybridClock initialized: now 1790330564527599 us; error 27 us; skew 500 ppm
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:44.530200 21441 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:44.530619 21441 webserver.cc:460] Webserver started at http://127.19.46.193:44857/ using document root <none> and password file <none>
I20260925 10:02:44.530864 21441 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:44.530934 21441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:44.531080 21441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:44.532141 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "dded8e7e5b5ca052ea22274e390a5434"
server_key_iv: "057dd93be64b88eeb0a997f05f869599"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.532498 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "dded8e7e5b5ca052ea22274e390a5434"
server_key_iv: "057dd93be64b88eeb0a997f05f869599"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.533857 21441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:44.534605 21456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.534772 21441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:44.534862 21441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "dded8e7e5b5ca052ea22274e390a5434"
server_key_iv: "057dd93be64b88eeb0a997f05f869599"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.534945 21441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:44.562930 21441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:44.564927 21441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:44.565109 21441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:44.565778 21441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:44.565835 21441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.565877 21441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:44.565905 21441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:44.572752 21441 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:42785
I20260925 10:02:44.572795 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:42785 every 8 connection(s)
I20260925 10:02:44.573167 21441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:44.575804 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.566226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:47359 (local address 127.19.46.254:46293)
0925 10:02:44.566272 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.566274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.566534 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:02:44.567068 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.567070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.567088 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.567131 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.568047 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.568937 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.570352 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.570735 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.570826 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.573075 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.573076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.573077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.573085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.574534 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.574904 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.574905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.574905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.574954 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.575214 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.575215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.575216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.575337 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.575402 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.575668 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.575742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:02:44.575903 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.566193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.193:47359)
0925 10:02:44.566356 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.566392 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.566944 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.567209 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.567213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.567606 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.567930 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.567937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.569065 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.569066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.570254 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.570257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.570853 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.570854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.570969 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.571511 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.571520 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.572981 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.574618 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.574624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.574625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.574838 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.575011 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.575011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.575011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.575134 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.575404 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.575408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.575515 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.575597 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.575714 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260925 10:02:44.576356 21572 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46293
I20260925 10:02:44.576436 21572 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:44.576646 21572 heartbeater.cc:511] Master 127.19.46.254:46293 requested a full tablet report, sending...
I20260925 10:02:44.577214 21368 ts_manager.cc:194] Registered new tserver with Master: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785)
I20260925 10:02:44.578052 21368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:47359
I20260925 10:02:44.578547 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 21441
I20260925 10:02:44.578639 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:44.578822 19643 external_mini_cluster.cc:1428] Setting key f7c7a45471768a78c0080d6413207e1e
I20260925 10:02:44.583966 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.577540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:49773 (local address 127.19.46.254:46293)
0925 10:02:44.577582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.577584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.577703 (+ 119us) server_negotiation.cc:407] Connection header received
0925 10:02:44.577766 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.577767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.577776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.577821 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.578143 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.578854 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.579666 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.579874 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.579960 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.581072 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.581073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.581073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.581080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.582796 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.583209 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.583210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.583210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.583264 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.583483 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.583484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.583484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.583594 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.583648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.583829 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.583896 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:02:44.584241 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.577498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.193:49773)
0925 10:02:44.577633 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.577643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.577707 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.577833 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.577834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.578008 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.578059 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.578060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.578942 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.578943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.579573 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.579576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.579963 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.579963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.580035 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.580507 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.580513 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.580930 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.582885 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.582886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.582887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.583130 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.583336 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.583337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.583337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.583413 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.583663 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.583665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.583723 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.583812 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.583938 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:44.606400 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46293
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:44.689250 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:44.689473 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:44.689499 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:44.690588 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:44.690629 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:44.690649 21581 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:02:44.690667 21581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:44.690685 21581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:44.692047 21581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:44.692238 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:44.692414 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:44.697300 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.697351 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.697535 21581 server_base.cc:1034] running on GCE node
W20260925 10:02:44.697296 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.697800 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:44.698056 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:44.699234 21581 hybrid_clock.cc:648] HybridClock initialized: now 1790330564699229 us; error 54 us; skew 500 ppm
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:44.701722 21581 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:44.702127 21581 webserver.cc:460] Webserver started at http://127.19.46.194:39375/ using document root <none> and password file <none>
I20260925 10:02:44.702332 21581 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:44.702373 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:44.702461 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:44.703497 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "dd1ed03f35e848b395e560fc47682e47"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "0013d850db701c6501f15a4e95481913"
server_key_iv: "a1c523639ca60bef695c1f9829a3dec2"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.703819 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "dd1ed03f35e848b395e560fc47682e47"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "0013d850db701c6501f15a4e95481913"
server_key_iv: "a1c523639ca60bef695c1f9829a3dec2"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.705119 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:02:44.706099 21596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.706319 21581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:44.706408 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "dd1ed03f35e848b395e560fc47682e47"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "0013d850db701c6501f15a4e95481913"
server_key_iv: "a1c523639ca60bef695c1f9829a3dec2"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.706465 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:44.742437 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:44.744431 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:44.744545 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:44.745013 21581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:44.745047 21581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.745071 21581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:44.745131 21581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.751545 21581 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:35429
I20260925 10:02:44.751694 21711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:35429 every 8 connection(s)
I20260925 10:02:44.751964 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:44.755642 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.745649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:40011 (local address 127.19.46.254:46293)
0925 10:02:44.745696 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.745699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.745942 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:02:44.746547 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.746549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.746566 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.746649 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.747541 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.748398 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.750077 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.750426 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.750521 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.753121 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.753122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.753123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.753130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.754403 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.754796 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.754798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.754798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.754857 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.755106 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.755107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.755107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.755219 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.755272 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.755504 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.755575 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:02:44.755854 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.745646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.194:40011)
0925 10:02:44.745820 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.745855 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.746383 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.746725 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.746729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.747113 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.747422 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.747429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.748508 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.748509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.749976 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.749978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.750523 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.750524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.750696 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.751286 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.751296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.753032 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.754479 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.754485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.754486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.754705 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.754925 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.754926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.754926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.755018 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.755277 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.755281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.755392 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.755473 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.755592 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260925 10:02:44.756491 21712 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46293
I20260925 10:02:44.756580 21712 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:44.756778 21712 heartbeater.cc:511] Master 127.19.46.254:46293 requested a full tablet report, sending...
I20260925 10:02:44.757200 21369 ts_manager.cc:194] Registered new tserver with Master: dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:44.757691 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:40011
I20260925 10:02:44.762219 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 21581
I20260925 10:02:44.762332 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:44.762558 19643 external_mini_cluster.cc:1428] Setting key 2a39f27af15a364f2bdb7064bf623339
I20260925 10:02:44.764361 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.757701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.194:40097)
0925 10:02:44.757867 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.757878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.757939 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.758410 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.758411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.758616 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.758676 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.758677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.759333 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.759334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.759881 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.759883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.760284 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.760285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.760366 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.760858 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.760863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.761196 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.763238 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.763239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.763240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.763453 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.763661 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.763662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.763662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.763736 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.764093 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.764094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.764137 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.764200 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.764286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260925 10:02:44.764463 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.757769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:40097 (local address 127.19.46.254:46293)
0925 10:02:44.757813 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.757815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.757944 (+ 129us) server_negotiation.cc:407] Connection header received
0925 10:02:44.758253 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.758254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.758270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.758340 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.758768 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.759231 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.759997 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.760197 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.760256 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.761377 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.761379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.761379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.761386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.763141 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.763526 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.763527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.763527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.763592 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.763801 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.763802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.763802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.764021 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.764094 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.764236 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.764322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:44.792876 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46293
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:44.881570 21721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:44.881796 21721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:44.881824 21721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:44.883006 21721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:44.883046 21721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:44.883068 21721 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:02:44.883080 21721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:02:44.883097 21721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:44.884526 21721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:44.884740 21721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:44.884925 21721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:44.890301 21729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.890304 21727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:44.890492 21726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:44.890718 21721 server_base.cc:1034] running on GCE node
I20260925 10:02:44.890955 21721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:44.891222 21721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:44.892395 21721 hybrid_clock.cc:648] HybridClock initialized: now 1790330564892394 us; error 47 us; skew 500 ppm
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:44.895179 21721 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:44.895718 21721 webserver.cc:460] Webserver started at http://127.19.46.195:44523/ using document root <none> and password file <none>
I20260925 10:02:44.895969 21721 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:44.896044 21721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:44.896214 21721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:44.897378 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "4ed8e27680004bcdbf063f6a0324ce03"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "b5da06858e13ace5fb89e61fca5f24a5"
server_key_iv: "21f84cece173d49773f3f4e028385ccf"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.897867 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "4ed8e27680004bcdbf063f6a0324ce03"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "b5da06858e13ace5fb89e61fca5f24a5"
server_key_iv: "21f84cece173d49773f3f4e028385ccf"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.899468 21721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:44.900301 21736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.900584 21721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:44.900679 21721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "4ed8e27680004bcdbf063f6a0324ce03"
format_stamp: "Formatted at 2026-09-25 10:02:44 on dist-test-slave-f9rl"
server_key: "b5da06858e13ace5fb89e61fca5f24a5"
server_key_iv: "21f84cece173d49773f3f4e028385ccf"
server_key_version: "encryptionkey@0"
I20260925 10:02:44.900743 21721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:44.927294 21721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:44.929809 21721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:44.929971 21721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:44.930549 21721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:44.930594 21721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.930630 21721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:44.930682 21721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:44.938565 21721 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:38579
I20260925 10:02:44.938737 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:38579 every 8 connection(s)
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:44.939036 21721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:44.942438 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.931152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:46791 (local address 127.19.46.254:46293)
0925 10:02:44.931219 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.931222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.931472 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:02:44.932083 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.932084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.932100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.932159 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.933183 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.934105 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.935975 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.936359 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.936422 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.939757 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.939759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.939760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.939768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.940999 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.941420 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.941422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.941422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.941505 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.941768 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.941769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.941769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.941899 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.941978 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.942257 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.942345 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:02:44.942632 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.931082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.195:46791)
0925 10:02:44.931342 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.931381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.931949 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.932240 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.932244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.932729 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.933080 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.933087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.934221 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.934222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.935864 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.935869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.936453 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.936454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.936643 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.937456 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.937478 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.939633 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.941089 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.941097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.941099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.941334 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.941575 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.941576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.941576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.941696 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.941968 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.941975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.942119 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.942233 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.942395 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260925 10:02:44.943084 21852 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46293
I20260925 10:02:44.943221 21852 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:44.943428 21852 heartbeater.cc:511] Master 127.19.46.254:46293 requested a full tablet report, sending...
I20260925 10:02:44.944033 21368 ts_manager.cc:194] Registered new tserver with Master: 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579)
I20260925 10:02:44.944633 21368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:46791
I20260925 10:02:44.948710 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 21721
I20260925 10:02:44.948807 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:44.948992 19643 external_mini_cluster.cc:1428] Setting key 9ff02cafa43986cfd1a3cc35e0750e8f
I20260925 10:02:44.949980 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:02:44.950403 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.944475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.19.46.195:33133)
0925 10:02:44.944601 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.944613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.944676 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.944867 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.944868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.945081 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:44.945179 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.945181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.945942 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.945943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.946427 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.946428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.946797 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.946797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.946852 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.947329 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:44.947334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:44.947622 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:44.949323 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.949324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.949325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.949624 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.949816 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:44.949817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:44.949817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:44.949887 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:44.950164 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:44.950165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:44.950202 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.950257 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.950336 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260925 10:02:44.950466 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.944543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:33133 (local address 127.19.46.254:46293)
0925 10:02:44.944589 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.944591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.944659 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:02:44.944754 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.944756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.944768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.944837 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.945265 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.945858 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.946529 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.946724 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.946786 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.947723 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.947724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.947725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.947730 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.949242 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.949700 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.949701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.949701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.949751 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.949968 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.949969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.949969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.950098 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.950171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.950295 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.950366 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:02:44 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:44.972927 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.966128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.19.46.254:46293)
0925 10:02:44.966197 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.966199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.966207 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:44.966234 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.966235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.966244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.966294 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:44.966846 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.967325 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.968041 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.968237 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.968285 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.970494 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:44.970495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:44.970495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:44.970501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:44.971762 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.972166 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.972168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.972168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.972229 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:44.972431 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:44.972432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:44.972433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:44.972549 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:44.972631 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.972750 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.972812 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:02:44.974891 21368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:44.976627 21368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:44.983445 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:38579 (local address 127.0.0.1:42138)
0925 10:02:44.979371 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.979414 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.979484 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.979679 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.979682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.979684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:44.979753 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.979755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.981036 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.981037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.982382 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.982385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983113 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.983204 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983217 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.983284 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.983335 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260925 10:02:44.983554 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.0.0.1:33008)
0925 10:02:44.979755 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.979766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.979829 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.980085 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.980086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.980088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:44.980141 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.980143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.981316 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.981317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.982433 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.982435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983281 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.983337 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983340 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.983397 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.983431 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260925 10:02:44.983556 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.19.46.194:35429)
0925 10:02:44.979747 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.979751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.979891 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:02:44.979935 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.979937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.979953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.980019 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:44.980201 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.981228 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.982507 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983218 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983282 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983329 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.983411 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.983453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260925 10:02:44.983897 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.19.46.193:42785)
0925 10:02:44.980265 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.980269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.980276 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:02:44.980305 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.980306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.980317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.980395 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:44.980502 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.981193 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.982389 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983145 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983218 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983275 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.983750 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.983794 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":97,"threads_started":1}
I20260925 10:02:44.984011 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.19.46.195:38579)
0925 10:02:44.979475 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:44.979479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:44.979492 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:02:44.979576 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:44.979578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:44.979593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:44.979667 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:44.979821 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.980939 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.982485 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983008 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983111 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983183 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:44.983880 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 10:02:44.983922 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260925 10:02:44.984189 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:44.979409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42785 (local address 127.0.0.1:49694)
0925 10:02:44.979789 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:44.979798 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:44.979861 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:44.980404 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:44.980405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:44.980407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:44.980451 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.980452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.981279 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.981280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.982303 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:44.982306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:44.983311 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:44.983312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:44.983375 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:44.983381 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:02:44.983431 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:02:44.983470 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":66,"thread_start_us":68,"threads_started":1}
I20260925 10:02:44.984275 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 (DEFAULT_TABLE table=test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:44.984664 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073ea74d83b44c22b8ae680fc88ffe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:44.984608 21786 tablet_service.cc:1467] Processing CreateTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 (DEFAULT_TABLE table=test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:44.984937 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073ea74d83b44c22b8ae680fc88ffe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:44.986089 21506 tablet_service.cc:1467] Processing CreateTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 (DEFAULT_TABLE table=test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:44.986395 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073ea74d83b44c22b8ae680fc88ffe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:44.988147 21875 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Bootstrap starting.
I20260925 10:02:44.988462 21876 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Bootstrap starting.
I20260925 10:02:44.989032 21876 tablet_bootstrap.cc:654] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:44.989358 21876 log.cc:824] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:44.989528 21875 tablet_bootstrap.cc:654] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:44.989569 21877 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Bootstrap starting.
I20260925 10:02:44.989987 21875 log.cc:824] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:44.990186 21876 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: No bootstrap required, opened a new log
I20260925 10:02:44.990173 21877 tablet_bootstrap.cc:654] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:44.990252 21876 ts_tablet_manager.cc:1397] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:02:44.990517 21877 log.cc:824] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:44.990841 21875 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: No bootstrap required, opened a new log
I20260925 10:02:44.990916 21875 ts_tablet_manager.cc:1397] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:02:44.991195 21877 tablet_bootstrap.cc:492] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: No bootstrap required, opened a new log
I20260925 10:02:44.991279 21877 ts_tablet_manager.cc:1397] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:02:44.992431 21876 raft_consensus.cc:348] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.992615 21876 raft_consensus.cc:374] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:44.992654 21876 raft_consensus.cc:729] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Initialized, Role: FOLLOWER
I20260925 10:02:44.992782 21876 consensus_queue.cc:260] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.993041 21876 ts_tablet_manager.cc:1428] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:02:44.992995 21877 raft_consensus.cc:348] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.992998 21875 raft_consensus.cc:348] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.993147 21877 raft_consensus.cc:374] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:44.993161 21712 heartbeater.cc:503] Master 127.19.46.254:46293 was elected leader, sending a full tablet report...
I20260925 10:02:44.993162 21875 raft_consensus.cc:374] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:44.993191 21877 raft_consensus.cc:729] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Initialized, Role: FOLLOWER
I20260925 10:02:44.993198 21875 raft_consensus.cc:729] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Initialized, Role: FOLLOWER
I20260925 10:02:44.993305 21875 consensus_queue.cc:260] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.993304 21877 consensus_queue.cc:260] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:44.993569 21875 ts_tablet_manager.cc:1428] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:02:44.993624 21877 ts_tablet_manager.cc:1428] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 10:02:44.993651 21572 heartbeater.cc:503] Master 127.19.46.254:46293 was elected leader, sending a full tablet report...
I20260925 10:02:44.993672 21852 heartbeater.cc:503] Master 127.19.46.254:46293 was elected leader, sending a full tablet report...
W20260925 10:02:45.002580 21713 tablet.cc:2369] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:45.029364 21881 raft_consensus.cc:482] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:45.029520 21881 raft_consensus.cc:504] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:45.029917 21881 leader_election.cc:288] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579), 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785)
I20260925 10:02:45.034191 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.030429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:43209 (local address 127.19.46.193:42785)
0925 10:02:45.030511 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.030513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.030551 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:02:45.030620 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.030620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.030628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.030707 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:45.030857 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.031628 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033174 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.033773 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033853 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.033863 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.034076 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.034119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:02:45.034191 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.030297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:47821 (local address 127.19.46.195:38579)
0925 10:02:45.030356 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.030358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.030371 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:02:45.030402 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.030403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.030414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.030519 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:45.030695 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.031608 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033174 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.033770 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033853 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.033863 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.034077 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.034128 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:02:45.034221 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.030183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:38579 (local address 127.19.46.194:47821)
0925 10:02:45.030247 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.030259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.030304 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.030524 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.030525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.030527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:45.030601 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.030603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.031719 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.031720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.033044 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.033048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.033888 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.033981 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.033986 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.034053 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.034104 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":50}
I20260925 10:02:45.034233 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.030314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42785 (local address 127.19.46.194:43209)
0925 10:02:45.030493 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.030503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.030560 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.030735 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.030736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.030737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:45.030783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.030784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.031719 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.031720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.033044 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.033048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.033888 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.033889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.033981 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.033986 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.034053 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.034133 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":116,"thread_start_us":120,"threads_started":1}
I20260925 10:02:45.034683 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03" is_pre_election: true
I20260925 10:02:45.034723 21883 raft_consensus.cc:482] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:45.034734 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" is_pre_election: true
I20260925 10:02:45.034826 21883 raft_consensus.cc:504] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:45.034864 21806 raft_consensus.cc:2454] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 0.
I20260925 10:02:45.034912 21526 raft_consensus.cc:2454] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 0.
I20260925 10:02:45.035084 21598 leader_election.cc:302] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed8e27680004bcdbf063f6a0324ce03, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.035148 21883 leader_election.cc:288] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579), dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:45.035533 21881 raft_consensus.cc:2781] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:45.035621 21881 raft_consensus.cc:482] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:45.035658 21881 raft_consensus.cc:3037] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.036777 21881 raft_consensus.cc:504] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:45.036922 21881 leader_election.cc:288] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 election: Requested vote from peers 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579), 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785)
I20260925 10:02:45.037106 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
I20260925 10:02:45.037091 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03"
I20260925 10:02:45.037173 21526 raft_consensus.cc:3037] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.037173 21806 raft_consensus.cc:3037] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.038118 21806 raft_consensus.cc:2454] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 1.
I20260925 10:02:45.038154 21526 raft_consensus.cc:2454] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 1.
I20260925 10:02:45.038306 21598 leader_election.cc:302] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.038470 21881 raft_consensus.cc:2781] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:45.038689 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.035242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:38579 (local address 127.19.46.193:51335)
0925 10:02:45.035289 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.035300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.035336 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.035534 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.035535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.035536 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:45.035597 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.035599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.036430 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.036431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.037646 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.037650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.038438 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.038439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.038515 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.038519 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.038567 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.038603 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":27}
I20260925 10:02:45.038712 21881 raft_consensus.cc:686] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 1 LEADER]: Becoming Leader. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Running, Role: LEADER
I20260925 10:02:45.038839 21881 consensus_queue.cc:237] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } }
I20260925 10:02:45.039177 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.035286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:51335 (local address 127.19.46.195:38579)
0925 10:02:45.035337 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.035339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.035347 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:45.035430 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.035431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.035440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.035488 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:45.035678 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.036337 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.037748 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.038250 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.038313 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.038323 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.039042 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.039083 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:02:45.039294 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.035273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.19.46.193:32989)
0925 10:02:45.035465 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.035488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.035560 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.035699 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.035700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.035702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:45.035771 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.035773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.036356 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.036357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.038240 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.038243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.039039 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.039040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.039109 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.039113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.039162 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.039202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":37,"thread_start_us":73,"threads_started":1}
I20260925 10:02:45.039332 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03" is_pre_election: true
I20260925 10:02:45.039418 21806 raft_consensus.cc:2379] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc in current term 1: Already voted for candidate dd1ed03f35e848b395e560fc47682e47 in this term.
I20260925 10:02:45.039580 21368 catalog_manager.cc:5495] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd1ed03f35e848b395e560fc47682e47 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } health_report { overall_health: UNKNOWN } } }
I20260925 10:02:45.040009 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.035454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:32989 (local address 127.19.46.194:35429)
0925 10:02:45.035500 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.035502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.035536 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:02:45.035620 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.035621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.035630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.035681 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:45.035823 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.036271 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.038337 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.038914 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.038954 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.038961 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.039935 (+ 974us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.039969 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:02:45.040278 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073ea74d83b44c22b8ae680fc88ffe26" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47" is_pre_election: true
I20260925 10:02:45.040879 21460 leader_election.cc:302] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc; no voters: 4ed8e27680004bcdbf063f6a0324ce03, dd1ed03f35e848b395e560fc47682e47
I20260925 10:02:45.041007 21883 raft_consensus.cc:2726] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 10:02:45.073964 21573 tablet.cc:2369] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:02:45 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330564, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:45.091424 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.082384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.19.46.254:46293)
0925 10:02:45.082460 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.082462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.082474 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:02:45.082568 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.082568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.082578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.082651 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:45.083270 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.083771 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.084576 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.084774 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.084839 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.087908 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:45.087910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:45.087910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:45.087917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:45.090137 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:45.090583 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:45.090584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:45.090585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:45.090649 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:45.090853 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:45.090854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:45.090854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:45.090978 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:45.091051 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.091217 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.091309 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:02:45.091496 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.082257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46293 (local address 127.0.0.1:54654)
0925 10:02:45.082419 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.082430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.082481 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.082715 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.082717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.083092 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:45.083144 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.083147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.083869 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.083870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.084457 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.084458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.084853 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.084854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.084898 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.085671 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:45.085679 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:45.087784 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:45.090260 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:45.090271 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:45.090274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:45.090492 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:45.090725 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:45.090726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:45.090726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:45.090789 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:45.091050 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:45.091060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:45.091106 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.091193 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.091365 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":54,"thread_start_us":86,"threads_started":1}
I20260925 10:02:45.094774 21368 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:54654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:02:45.095026 21368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:45.096698 21786 tablet_service.cc:1467] Processing CreateTablet for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6fe1d05dd6140b5ab2002742293ae16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:45.096675 21506 tablet_service.cc:1467] Processing CreateTablet for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6fe1d05dd6140b5ab2002742293ae16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:45.096854 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.096853 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.098204 21646 tablet_service.cc:1467] Processing CreateTablet for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6fe1d05dd6140b5ab2002742293ae16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:02:45.098388 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8dc6d9e0334c2fae10f7e7b8d260ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.098483 21877 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Bootstrap starting.
I20260925 10:02:45.098493 21875 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: Bootstrap starting.
I20260925 10:02:45.098922 21877 tablet_bootstrap.cc:654] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.099114 21875 tablet_bootstrap.cc:654] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.099697 21877 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: No bootstrap required, opened a new log
I20260925 10:02:45.099761 21877 ts_tablet_manager.cc:1397] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:45.099923 21877 raft_consensus.cc:348] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.100014 21877 raft_consensus.cc:374] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.100037 21877 raft_consensus.cc:729] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.100080 21877 consensus_queue.cc:260] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.100116 21876 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: Bootstrap starting.
I20260925 10:02:45.100199 21877 ts_tablet_manager.cc:1428] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:45.100353 21883 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.100425 21883 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.100618 21876 tablet_bootstrap.cc:654] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.100656 21875 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: No bootstrap required, opened a new log
I20260925 10:02:45.100706 21875 ts_tablet_manager.cc:1397] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:02:45.100863 21875 raft_consensus.cc:348] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.100939 21875 raft_consensus.cc:374] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.100960 21875 raft_consensus.cc:729] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.101011 21875 consensus_queue.cc:260] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.101089 21875 ts_tablet_manager.cc:1428] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:02:45.101245 21882 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.101341 21882 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.101373 21876 tablet_bootstrap.cc:492] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: No bootstrap required, opened a new log
I20260925 10:02:45.101414 21876 ts_tablet_manager.cc:1397] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:45.101583 21876 raft_consensus.cc:348] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.101692 21876 raft_consensus.cc:374] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.101727 21876 raft_consensus.cc:729] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.101780 21876 consensus_queue.cc:260] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.101871 21876 ts_tablet_manager.cc:1428] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:45.101987 21881 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.102080 21881 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 10:02:45.189746 21853 tablet.cc:2369] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:45.290697 21881 raft_consensus.cc:482] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:45.290803 21881 raft_consensus.cc:504] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.290951 21881 leader_election.cc:288] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579)
I20260925 10:02:45.291265 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8dc6d9e0334c2fae10f7e7b8d260ce" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" is_pre_election: true
I20260925 10:02:45.291401 21526 raft_consensus.cc:2454] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 0.
I20260925 10:02:45.291342 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8dc6d9e0334c2fae10f7e7b8d260ce" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03" is_pre_election: true
I20260925 10:02:45.291447 21806 raft_consensus.cc:2454] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 0.
I20260925 10:02:45.291613 21598 leader_election.cc:302] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.291714 21881 raft_consensus.cc:2781] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:45.291760 21881 raft_consensus.cc:482] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:45.291800 21881 raft_consensus.cc:3037] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.292495 21881 raft_consensus.cc:504] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.292641 21881 leader_election.cc:288] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579)
I20260925 10:02:45.292845 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8dc6d9e0334c2fae10f7e7b8d260ce" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
I20260925 10:02:45.292858 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8dc6d9e0334c2fae10f7e7b8d260ce" candidate_uuid: "dd1ed03f35e848b395e560fc47682e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03"
I20260925 10:02:45.292930 21526 raft_consensus.cc:3037] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.292953 21806 raft_consensus.cc:3037] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.293732 21526 raft_consensus.cc:2454] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 1.
I20260925 10:02:45.293831 21806 raft_consensus.cc:2454] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1ed03f35e848b395e560fc47682e47 in term 1.
I20260925 10:02:45.293924 21598 leader_election.cc:302] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.294050 21881 raft_consensus.cc:2781] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:45.294131 21881 raft_consensus.cc:686] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [term 1 LEADER]: Becoming Leader. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Running, Role: LEADER
I20260925 10:02:45.294199 21881 consensus_queue.cc:237] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.294517 21886 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: TxnStatusTablet state changed. Reason: New leader dd1ed03f35e848b395e560fc47682e47. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.294633 21886 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:45.294808 21898 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:02:45.294950 21898 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:02:45.294870 21369 catalog_manager.cc:5495] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd1ed03f35e848b395e560fc47682e47 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20260925 10:02:45.334442 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.331441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.19.46.194:35429)
0925 10:02:45.331502 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.331504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.331512 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:45.331550 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.331551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.331576 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.331656 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.331814 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.332527 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.333474 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.333790 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.333876 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.334071 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.334196 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.334292 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.334334 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.334356 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:02:45.334450 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.331207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.0.0.1:33018)
0925 10:02:45.331293 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.331304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.331347 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.331665 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.331666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.331667 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:45.331735 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.331737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.332649 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.332650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.333336 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.333339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.333882 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.333883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.333952 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.333968 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.334267 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.334277 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.334327 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.334379 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:02:45.336721 21806 raft_consensus.cc:1261] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Refusing update from remote peer dd1ed03f35e848b395e560fc47682e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.336781 21526 raft_consensus.cc:1261] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Refusing update from remote peer dd1ed03f35e848b395e560fc47682e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.337102 21886 consensus_queue.cc:1035] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.337261 21881 consensus_queue.cc:1035] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.338550 21883 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: TxnStatusTablet state changed. Reason: New leader dd1ed03f35e848b395e560fc47682e47. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.338668 21883 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.338611 21882 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: TxnStatusTablet state changed. Reason: New leader dd1ed03f35e848b395e560fc47682e47. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.338729 21882 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.339850 21881 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.339944 21881 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:45.340008 21882 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.340088 21882 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 4ed8e27680004bcdbf063f6a0324ce03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.340687 21901 mvcc.cc:205] Tried to move back new op lower bound from 7333193995616374784 to 7333193995445547008. Current Snapshot: MvccSnapshot[applied={T|T < 7333193995616374784 or (T in {7333193995616374784})}]
I20260925 10:02:45.340885 21903 mvcc.cc:205] Tried to move back new op lower bound from 7333193995616374784 to 7333193995445547008. Current Snapshot: MvccSnapshot[applied={T|T < 7333193995616374784}]
I20260925 10:02:45.341471 21881 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.341569 21881 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P dd1ed03f35e848b395e560fc47682e47: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:02:45.341895 21883 tablet_replica.cc:439] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd1ed03f35e848b395e560fc47682e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } } }
I20260925 10:02:45.342007 21883 tablet_replica.cc:442] T aa8dc6d9e0334c2fae10f7e7b8d260ce P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:02:45.344452 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.342000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.19.46.194:35429)
0925 10:02:45.342104 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.342106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.342113 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:02:45.342135 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.342136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.342146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.342208 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.342342 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.342914 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.343776 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.343987 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.344066 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.344195 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.344229 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.344278 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.344353 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.344372 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:02:45.350634 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.347521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:45455 (local address 127.19.46.194:35429)
0925 10:02:45.347587 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.347589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.347699 (+ 110us) server_negotiation.cc:407] Connection header received
0925 10:02:45.347762 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.347763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.347772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.347823 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.347992 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.348509 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.349731 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.349929 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.350003 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.350279 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.350309 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.350362 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.350445 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.350554 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":42}
I20260925 10:02:45.350749 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.347409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.19.46.194:45455)
0925 10:02:45.347606 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.347618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.347701 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.347834 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.347834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.347836 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:45.347929 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.347931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.348614 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.348616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.349618 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.349621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.350054 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.350055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.350126 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.350143 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.350381 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.350383 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.350443 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.350587 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260925 10:02:45.353467 21526 raft_consensus.cc:1261] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Refusing update from remote peer dd1ed03f35e848b395e560fc47682e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.353482 21806 raft_consensus.cc:1261] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Refusing update from remote peer dd1ed03f35e848b395e560fc47682e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.353708 21886 consensus_queue.cc:1035] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.353925 21913 consensus_queue.cc:1035] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.356747 21902 mvcc.cc:205] Tried to move back new op lower bound from 7333193995686367232 to 7333193994399625216. Current Snapshot: MvccSnapshot[applied={T|T < 7333193995686367232}]
I20260925 10:02:45.369256 21368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:45.369367 21368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:45.370177 21368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Sending DeleteTablet for 3 replicas of tablet 073ea74d83b44c22b8ae680fc88ffe26
I20260925 10:02:45.370615 21786 tablet_service.cc:1514] Processing DeleteTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:45 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260925 10:02:45.370627 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:45 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:33008
I20260925 10:02:45.370880 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet 073ea74d83b44c22b8ae680fc88ffe26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:45 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 10:02:45.371059 21919 tablet_replica.cc:330] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: stopping tablet replica
I20260925 10:02:45.371189 21919 raft_consensus.cc:2229] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:02:45.371191 21920 tablet_replica.cc:330] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: stopping tablet replica
I20260925 10:02:45.371310 21919 raft_consensus.cc:2258] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:45.371338 21368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:45.371492 21368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:45.371420 21920 raft_consensus.cc:2229] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:45.371769 21920 raft_consensus.cc:2258] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:45.371876 21918 tablet_replica.cc:330] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: stopping tablet replica
I20260925 10:02:45.371994 21918 raft_consensus.cc:2229] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:45.372066 21918 raft_consensus.cc:2258] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:45.372344 21920 ts_tablet_manager.cc:1905] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:45.372457 21918 ts_tablet_manager.cc:1905] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:45.373322 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 20470786f5924fd48688786d3099c20b (DEFAULT_TABLE table=test-table [id=e13d279edf464b2ba16cb5770ddac7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:45.373486 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20470786f5924fd48688786d3099c20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.374429 21918 ts_tablet_manager.cc:1918] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:45.374513 21918 log.cc:1196] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/073ea74d83b44c22b8ae680fc88ffe26
I20260925 10:02:45.374615 21920 ts_tablet_manager.cc:1918] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:45.374748 21920 log.cc:1196] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/073ea74d83b44c22b8ae680fc88ffe26
I20260925 10:02:45.374804 21918 ts_tablet_manager.cc:1939] T 073ea74d83b44c22b8ae680fc88ffe26 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting consensus metadata
I20260925 10:02:45.375083 21876 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Bootstrap starting.
I20260925 10:02:45.375203 21920 ts_tablet_manager.cc:1939] T 073ea74d83b44c22b8ae680fc88ffe26 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting consensus metadata
I20260925 10:02:45.375501 21876 tablet_bootstrap.cc:654] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.376096 21876 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: No bootstrap required, opened a new log
I20260925 10:02:45.376102 21354 catalog_manager.cc:4841] TS 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785): tablet 073ea74d83b44c22b8ae680fc88ffe26 (table test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]) successfully deleted
I20260925 10:02:45.376133 21506 tablet_service.cc:1467] Processing CreateTablet for tablet 20470786f5924fd48688786d3099c20b (DEFAULT_TABLE table=test-table [id=e13d279edf464b2ba16cb5770ddac7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:45.376151 21876 ts_tablet_manager.cc:1397] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:45.376189 21354 catalog_manager.cc:4841] TS 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579): tablet 073ea74d83b44c22b8ae680fc88ffe26 (table test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]) successfully deleted
I20260925 10:02:45.376236 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20470786f5924fd48688786d3099c20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.376273 21876 raft_consensus.cc:348] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.376345 21876 raft_consensus.cc:374] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.376367 21876 raft_consensus.cc:729] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.376362 21786 tablet_service.cc:1467] Processing CreateTablet for tablet 20470786f5924fd48688786d3099c20b (DEFAULT_TABLE table=test-table [id=e13d279edf464b2ba16cb5770ddac7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:45.376415 21876 consensus_queue.cc:260] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.376463 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20470786f5924fd48688786d3099c20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:45.376516 21876 ts_tablet_manager.cc:1428] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:45.377593 21877 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Bootstrap starting.
I20260925 10:02:45.377950 21875 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Bootstrap starting.
I20260925 10:02:45.378016 21877 tablet_bootstrap.cc:654] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.378316 21919 ts_tablet_manager.cc:1905] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:45.378402 21875 tablet_bootstrap.cc:654] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:45.380306 21875 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: No bootstrap required, opened a new log
I20260925 10:02:45.380357 21875 ts_tablet_manager.cc:1397] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:02:45.380498 21875 raft_consensus.cc:348] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.380565 21875 raft_consensus.cc:374] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.380586 21875 raft_consensus.cc:729] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.380589 21919 ts_tablet_manager.cc:1918] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:45.380668 21919 log.cc:1196] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/073ea74d83b44c22b8ae680fc88ffe26
I20260925 10:02:45.380650 21875 consensus_queue.cc:260] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.380749 21875 ts_tablet_manager.cc:1428] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:45.380975 21919 ts_tablet_manager.cc:1939] T 073ea74d83b44c22b8ae680fc88ffe26 P dd1ed03f35e848b395e560fc47682e47: Deleting consensus metadata
I20260925 10:02:45.381608 21356 catalog_manager.cc:4841] TS dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429): tablet 073ea74d83b44c22b8ae680fc88ffe26 (table test-table [id=c521e9948c504862bcc7cbe58c8fd7bf]) successfully deleted
I20260925 10:02:45.381865 21877 tablet_bootstrap.cc:492] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: No bootstrap required, opened a new log
I20260925 10:02:45.381913 21877 ts_tablet_manager.cc:1397] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260925 10:02:45.382033 21877 raft_consensus.cc:348] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.382091 21877 raft_consensus.cc:374] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:45.382107 21877 raft_consensus.cc:729] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Initialized, Role: FOLLOWER
I20260925 10:02:45.382171 21877 consensus_queue.cc:260] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.382292 21877 ts_tablet_manager.cc:1428] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:45.412523 21883 raft_consensus.cc:482] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:45.412631 21883 raft_consensus.cc:504] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.412817 21883 leader_election.cc:288] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429), 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579)
I20260925 10:02:45.413148 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20470786f5924fd48688786d3099c20b" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47" is_pre_election: true
I20260925 10:02:45.413177 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20470786f5924fd48688786d3099c20b" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03" is_pre_election: true
I20260925 10:02:45.413268 21806 raft_consensus.cc:2454] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc in term 0.
I20260925 10:02:45.413276 21666 raft_consensus.cc:2454] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc in term 0.
I20260925 10:02:45.413481 21460 leader_election.cc:302] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.413604 21883 raft_consensus.cc:2781] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:45.413635 21883 raft_consensus.cc:482] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:45.413658 21883 raft_consensus.cc:3037] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.414440 21883 raft_consensus.cc:504] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.414623 21883 leader_election.cc:288] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 election: Requested vote from peers dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429), 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579)
I20260925 10:02:45.414808 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20470786f5924fd48688786d3099c20b" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47"
I20260925 10:02:45.414918 21666 raft_consensus.cc:3037] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.414963 21806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20470786f5924fd48688786d3099c20b" candidate_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed8e27680004bcdbf063f6a0324ce03"
I20260925 10:02:45.415071 21806 raft_consensus.cc:3037] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:45.415674 21666 raft_consensus.cc:2454] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc in term 1.
I20260925 10:02:45.415859 21460 leader_election.cc:302] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:45.415968 21883 raft_consensus.cc:2781] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:45.416006 21806 raft_consensus.cc:2454] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc in term 1.
I20260925 10:02:45.416080 21883 raft_consensus.cc:686] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Running, Role: LEADER
I20260925 10:02:45.416149 21883 consensus_queue.cc:237] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:45.417032 21368 catalog_manager.cc:5495] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193). New cstate: current_term: 1 leader_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20260925 10:02:45.430704 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.427839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.19.46.193:42785)
0925 10:02:45.427887 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.427889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.427900 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:45.427980 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.427981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.428004 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.428163 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.428322 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.428932 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.429825 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.430057 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.430142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.430320 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.430440 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.430524 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.430584 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.430606 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:02:45.436180 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.433465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:57349 (local address 127.19.46.194:35429)
0925 10:02:45.433517 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.433534 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.433590 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:02:45.433665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.433666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.433675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.433723 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.433861 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.434354 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.435474 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.435684 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.435746 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.435951 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.435981 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.436024 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.436097 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.436114 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:02:45.436251 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.433343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.19.46.193:57349)
0925 10:02:45.433532 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.433544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.433607 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.433727 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.433728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.433730 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:45.433800 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.433802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.434437 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.434438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.435378 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.435381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.435779 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.435780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.435845 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.435865 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.436039 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.436041 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.436093 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.436137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260925 10:02:45.438869 21806 raft_consensus.cc:1261] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Refusing update from remote peer 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.438983 21666 raft_consensus.cc:1261] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Refusing update from remote peer 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:45.439083 21883 consensus_queue.cc:1035] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.439435 21924 consensus_queue.cc:1035] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:45.447743 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.444815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39741 (local address 127.19.46.193:42785)
0925 10:02:45.444977 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:45.444979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:45.444987 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:45.445009 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:45.445010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:45.445020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:45.445075 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:45.445245 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.446008 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.446929 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.447268 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.447327 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.447483 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.447519 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.447570 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:45.447634 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:02:45.447655 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 10:02:45.447769 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:45.444711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42785 (local address 127.19.46.194:39741)
0925 10:02:45.444801 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:45.444813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:45.444851 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:45.445095 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:45.445096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:45.445098 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:45.445151 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.445153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.446115 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.446117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.446806 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:45.446809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:45.447338 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:45.447340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:45.447395 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:45.447402 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:45.447587 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:45.447589 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:45.447632 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:02:45.447686 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 10:02:45.448275 21902 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:02:46.974256 21367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:46.974378 21367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:46.975143 21367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Sending DeleteTablet for 3 replicas of tablet 20470786f5924fd48688786d3099c20b
I20260925 10:02:46.975482 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet 20470786f5924fd48688786d3099c20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:46 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 10:02:46.975489 21786 tablet_service.cc:1514] Processing DeleteTablet for tablet 20470786f5924fd48688786d3099c20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:46 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260925 10:02:46.975698 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 20470786f5924fd48688786d3099c20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:46 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:33008
I20260925 10:02:46.976032 21958 tablet_replica.cc:330] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: stopping tablet replica
I20260925 10:02:46.976137 21958 raft_consensus.cc:2229] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:46.976212 21958 raft_consensus.cc:2258] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:46.976511 21958 ts_tablet_manager.cc:1905] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:46.978299 21957 tablet_replica.cc:330] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: stopping tablet replica
I20260925 10:02:46.978380 21957 raft_consensus.cc:2229] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:02:46.978477 21957 raft_consensus.cc:2258] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:46.978796 21958 ts_tablet_manager.cc:1918] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:02:46.978881 21958 log.cc:1196] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/20470786f5924fd48688786d3099c20b
I20260925 10:02:46.979089 21958 ts_tablet_manager.cc:1939] T 20470786f5924fd48688786d3099c20b P dd1ed03f35e848b395e560fc47682e47: Deleting consensus metadata
I20260925 10:02:46.979614 21356 catalog_manager.cc:4841] TS dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429): tablet 20470786f5924fd48688786d3099c20b (table test-table [id=e13d279edf464b2ba16cb5770ddac7bd]) successfully deleted
I20260925 10:02:46.980862 21956 tablet_replica.cc:330] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: stopping tablet replica
I20260925 10:02:46.980938 21956 raft_consensus.cc:2229] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:46.980995 21956 raft_consensus.cc:2258] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:46.981244 21957 ts_tablet_manager.cc:1905] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:46.981262 21956 ts_tablet_manager.cc:1905] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:46.982949 21956 ts_tablet_manager.cc:1918] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:02:46.983014 21956 log.cc:1196] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/20470786f5924fd48688786d3099c20b
I20260925 10:02:46.983022 21957 ts_tablet_manager.cc:1918] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:02:46.983078 21957 log.cc:1196] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/20470786f5924fd48688786d3099c20b
I20260925 10:02:46.983245 21956 ts_tablet_manager.cc:1939] T 20470786f5924fd48688786d3099c20b P 4ed8e27680004bcdbf063f6a0324ce03: Deleting consensus metadata
I20260925 10:02:46.983341 21957 ts_tablet_manager.cc:1939] T 20470786f5924fd48688786d3099c20b P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting consensus metadata
I20260925 10:02:46.983683 21354 catalog_manager.cc:4841] TS 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785): tablet 20470786f5924fd48688786d3099c20b (table test-table [id=e13d279edf464b2ba16cb5770ddac7bd]) successfully deleted
I20260925 10:02:46.983791 21354 catalog_manager.cc:4841] TS 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579): tablet 20470786f5924fd48688786d3099c20b (table test-table [id=e13d279edf464b2ba16cb5770ddac7bd]) successfully deleted
Sep 25 10:02:46 dist-test-slave-f9rl krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:02:47 dist-test-slave-f9rl krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:47.006493 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:46.992696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.19.46.254:46293)
0925 10:02:46.992856 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:46.992858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:46.992868 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:46.992897 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:46.992897 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:46.992908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:46.992987 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:46.993905 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:46.994455 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:46.995307 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:46.995515 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:46.995579 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.003114 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:47.003149 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:47.003149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:47.003161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:47.005123 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:47.005683 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:47.005685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:47.005685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:47.005755 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:47.005968 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:47.005969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:47.005970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:47.006139 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:47.006228 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.006327 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.006409 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260925 10:02:47.008514 21367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:47.008734 21367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:47.010407 21506 tablet_service.cc:1467] Processing CreateTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 (DEFAULT_TABLE table=test-table [id=fa17ffd8796542a09111b2d989b8cbb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.010407 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 (DEFAULT_TABLE table=test-table [id=fa17ffd8796542a09111b2d989b8cbb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.010417 21786 tablet_service.cc:1467] Processing CreateTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 (DEFAULT_TABLE table=test-table [id=fa17ffd8796542a09111b2d989b8cbb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.010566 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1538db8f77f14ee3b3258f7bea6db907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.010576 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1538db8f77f14ee3b3258f7bea6db907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.010584 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1538db8f77f14ee3b3258f7bea6db907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.011732 21967 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Bootstrap starting.
I20260925 10:02:47.012116 21969 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Bootstrap starting.
I20260925 10:02:47.012459 21969 tablet_bootstrap.cc:654] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.012741 21967 tablet_bootstrap.cc:654] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.012774 21968 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Bootstrap starting.
I20260925 10:02:47.013074 21969 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: No bootstrap required, opened a new log
I20260925 10:02:47.013134 21969 ts_tablet_manager.cc:1397] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:47.013195 21968 tablet_bootstrap.cc:654] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.013247 21969 raft_consensus.cc:348] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.013339 21969 raft_consensus.cc:374] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.013367 21969 raft_consensus.cc:729] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.013417 21969 consensus_queue.cc:260] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.013542 21969 ts_tablet_manager.cc:1428] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:02:47.013767 21967 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: No bootstrap required, opened a new log
I20260925 10:02:47.013821 21967 ts_tablet_manager.cc:1397] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:02:47.013950 21967 raft_consensus.cc:348] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.014042 21967 raft_consensus.cc:374] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.014066 21967 raft_consensus.cc:729] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.014117 21967 consensus_queue.cc:260] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.014322 21967 ts_tablet_manager.cc:1428] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:47.014803 21968 tablet_bootstrap.cc:492] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: No bootstrap required, opened a new log
I20260925 10:02:47.014868 21968 ts_tablet_manager.cc:1397] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:02:47.014999 21968 raft_consensus.cc:348] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.015112 21968 raft_consensus.cc:374] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.015151 21968 raft_consensus.cc:729] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.015213 21968 consensus_queue.cc:260] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.015324 21968 ts_tablet_manager.cc:1428] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:47.165858 21973 raft_consensus.cc:482] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:47.165963 21973 raft_consensus.cc:504] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.166371 21973 leader_election.cc:288] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:47.169489 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.166628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:52347 (local address 127.19.46.193:42785)
0925 10:02:47.166768 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:47.166771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:47.166832 (+ 61us) server_negotiation.cc:407] Connection header received
0925 10:02:47.166961 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:47.166962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:47.166970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:47.167054 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:47.167238 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.167712 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.168723 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.169107 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.169193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.169202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.169344 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.169387 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 10:02:47.169495 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.166551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42785 (local address 127.19.46.195:52347)
0925 10:02:47.166763 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:47.166777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:47.166848 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:47.167080 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:47.167080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:47.167083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:47.167143 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.167144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.167803 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.167804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.168627 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.168629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.169195 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.169196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.169269 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.169273 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:47.169315 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 10:02:47.169378 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":30,"thread_start_us":118,"threads_started":1}
I20260925 10:02:47.169512 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.166579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.19.46.195:45965)
0925 10:02:47.166774 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:47.166784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:47.166843 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:47.167037 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:47.167038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:47.167041 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:47.167125 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.167127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.167804 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.167805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.168625 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.168627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.169192 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.169193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.169271 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.169280 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:02:47.169352 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:02:47.169390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":14,"thread_start_us":112,"threads_started":1}
I20260925 10:02:47.169516 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.166668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:45965 (local address 127.19.46.194:35429)
0925 10:02:47.166808 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:47.166810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:47.166824 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:02:47.166925 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:47.166926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:47.166939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:47.167016 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:47.167201 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.167711 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.168721 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.169103 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.169193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.169202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.169371 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.169409 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 10:02:47.169864 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1538db8f77f14ee3b3258f7bea6db907" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" is_pre_election: true
I20260925 10:02:47.169875 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1538db8f77f14ee3b3258f7bea6db907" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47" is_pre_election: true
I20260925 10:02:47.169960 21666 raft_consensus.cc:2454] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 0.
I20260925 10:02:47.169979 21526 raft_consensus.cc:2454] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 0.
I20260925 10:02:47.170171 21738 leader_election.cc:302] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, 4ed8e27680004bcdbf063f6a0324ce03; no voters:
I20260925 10:02:47.170326 21973 raft_consensus.cc:2781] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:47.170367 21973 raft_consensus.cc:482] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:47.170397 21973 raft_consensus.cc:3037] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.171098 21973 raft_consensus.cc:504] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.171254 21973 leader_election.cc:288] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:47.171382 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1538db8f77f14ee3b3258f7bea6db907" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
I20260925 10:02:47.171458 21526 raft_consensus.cc:3037] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.171496 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1538db8f77f14ee3b3258f7bea6db907" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47"
I20260925 10:02:47.171588 21666 raft_consensus.cc:3037] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.172163 21526 raft_consensus.cc:2454] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 1.
I20260925 10:02:47.172245 21666 raft_consensus.cc:2454] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 1.
I20260925 10:02:47.172329 21738 leader_election.cc:302] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, 4ed8e27680004bcdbf063f6a0324ce03; no voters:
I20260925 10:02:47.172405 21973 raft_consensus.cc:2781] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:47.172562 21973 raft_consensus.cc:686] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Running, Role: LEADER
I20260925 10:02:47.172649 21973 consensus_queue.cc:237] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.173465 21368 catalog_manager.cc:5495] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "4ed8e27680004bcdbf063f6a0324ce03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } health_report { overall_health: HEALTHY } } }
I20260925 10:02:47.201622 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.198820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.19.46.195:38579)
0925 10:02:47.198994 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:47.198996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:47.199002 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:02:47.199036 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:47.199037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:47.199057 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:47.199124 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:47.199347 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.199867 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.200725 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.200927 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.200984 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.201193 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:47.201303 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:47.201361 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.201478 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.201501 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260925 10:02:47.206861 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.204507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:60529 (local address 127.19.46.194:35429)
0925 10:02:47.204588 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:47.204590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:47.204601 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:47.204629 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:47.204630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:47.204641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:47.204717 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:47.204864 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.205328 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.206119 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.206327 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.206407 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.206579 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:47.206619 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:47.206688 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.206753 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.206776 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:02:47.206876 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.204342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.19.46.195:60529)
0925 10:02:47.204502 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:47.204511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:47.204577 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:47.204722 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:47.204723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:47.204724 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:47.204794 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.204795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.205414 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.205415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.206016 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.206018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.206421 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.206422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.206481 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.206494 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:47.206701 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:47.206703 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:47.206741 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 10:02:47.206787 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260925 10:02:47.209482 21666 raft_consensus.cc:1261] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Refusing update from remote peer 4ed8e27680004bcdbf063f6a0324ce03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:47.209506 21526 raft_consensus.cc:1261] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Refusing update from remote peer 4ed8e27680004bcdbf063f6a0324ce03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:47.209761 21973 consensus_queue.cc:1035] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:47.209934 21978 consensus_queue.cc:1035] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:47.217958 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.215250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:33149 (local address 127.19.46.195:38579)
0925 10:02:47.215291 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:47.215293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:47.215452 (+ 159us) server_negotiation.cc:407] Connection header received
0925 10:02:47.215510 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:47.215510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:47.215520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:47.215561 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:02:47.215659 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.216117 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.217123 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.217478 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.217551 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.217717 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:02:47.217749 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:02:47.217799 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:47.217865 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:02:47.217884 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:02:47.217984 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:47.215195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:38579 (local address 127.19.46.194:33149)
0925 10:02:47.215373 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:47.215386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:47.215449 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:47.215564 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:47.215565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:47.215566 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:02:47.215613 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.215615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.216208 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.216209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.217009 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:47.217032 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:47.217552 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:47.217553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:47.217611 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:47.217616 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:02:47.217808 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:02:47.217810 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:02:47.217860 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:02:47.217906 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260925 10:02:47.225153 21367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:47.225296 21367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:47.226038 21367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Sending DeleteTablet for 3 replicas of tablet 1538db8f77f14ee3b3258f7bea6db907
I20260925 10:02:47.226330 21786 tablet_service.cc:1514] Processing DeleteTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:47 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260925 10:02:47.226393 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:47 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:33008
I20260925 10:02:47.226399 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet 1538db8f77f14ee3b3258f7bea6db907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:47 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 10:02:47.226451 21956 tablet_replica.cc:330] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: stopping tablet replica
I20260925 10:02:47.226487 21958 tablet_replica.cc:330] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: stopping tablet replica
I20260925 10:02:47.226511 21956 raft_consensus.cc:2229] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:02:47.226534 21958 raft_consensus.cc:2229] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:47.226572 21958 raft_consensus.cc:2258] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:47.226647 21957 tablet_replica.cc:330] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: stopping tablet replica
I20260925 10:02:47.226774 21957 raft_consensus.cc:2229] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:47.226778 21956 raft_consensus.cc:2258] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:47.226822 21957 raft_consensus.cc:2258] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:47.227133 21956 ts_tablet_manager.cc:1905] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:47.227134 21957 ts_tablet_manager.cc:1905] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:47.227231 21958 ts_tablet_manager.cc:1905] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:47.229029 21958 ts_tablet_manager.cc:1918] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:47.228943 21367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:02:47.229085 21958 log.cc:1196] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/1538db8f77f14ee3b3258f7bea6db907
W20260925 10:02:47.229116 21367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:47.229115 21956 ts_tablet_manager.cc:1918] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:47.229175 21956 log.cc:1196] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/1538db8f77f14ee3b3258f7bea6db907
I20260925 10:02:47.229321 21958 ts_tablet_manager.cc:1939] T 1538db8f77f14ee3b3258f7bea6db907 P dd1ed03f35e848b395e560fc47682e47: Deleting consensus metadata
I20260925 10:02:47.229506 21956 ts_tablet_manager.cc:1939] T 1538db8f77f14ee3b3258f7bea6db907 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting consensus metadata
I20260925 10:02:47.229537 21957 ts_tablet_manager.cc:1918] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:02:47.229586 21957 log.cc:1196] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/1538db8f77f14ee3b3258f7bea6db907
I20260925 10:02:47.229799 21957 ts_tablet_manager.cc:1939] T 1538db8f77f14ee3b3258f7bea6db907 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Deleting consensus metadata
I20260925 10:02:47.230137 21354 catalog_manager.cc:4841] TS 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785): tablet 1538db8f77f14ee3b3258f7bea6db907 (table test-table [id=fa17ffd8796542a09111b2d989b8cbb2]) successfully deleted
I20260925 10:02:47.230336 21356 catalog_manager.cc:4841] TS dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429): tablet 1538db8f77f14ee3b3258f7bea6db907 (table test-table [id=fa17ffd8796542a09111b2d989b8cbb2]) successfully deleted
I20260925 10:02:47.230541 21506 tablet_service.cc:1467] Processing CreateTablet for tablet 2017d4715ce748698d04201e2ace4756 (DEFAULT_TABLE table=test-table [id=9bcb512b0d6b4eca859011012794dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.230541 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 2017d4715ce748698d04201e2ace4756 (DEFAULT_TABLE table=test-table [id=9bcb512b0d6b4eca859011012794dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.230664 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2017d4715ce748698d04201e2ace4756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.230664 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2017d4715ce748698d04201e2ace4756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.231889 21969 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: Bootstrap starting.
I20260925 10:02:47.232085 21354 catalog_manager.cc:4841] TS 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579): tablet 1538db8f77f14ee3b3258f7bea6db907 (table test-table [id=fa17ffd8796542a09111b2d989b8cbb2]) successfully deleted
I20260925 10:02:47.232169 21786 tablet_service.cc:1467] Processing CreateTablet for tablet 2017d4715ce748698d04201e2ace4756 (DEFAULT_TABLE table=test-table [id=9bcb512b0d6b4eca859011012794dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:47.232280 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2017d4715ce748698d04201e2ace4756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:47.232275 21969 tablet_bootstrap.cc:654] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.232584 21968 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Bootstrap starting.
I20260925 10:02:47.233012 21968 tablet_bootstrap.cc:654] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.233283 21967 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Bootstrap starting.
I20260925 10:02:47.233422 21969 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: No bootstrap required, opened a new log
I20260925 10:02:47.233597 21969 ts_tablet_manager.cc:1397] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:02:47.233697 21967 tablet_bootstrap.cc:654] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:47.233721 21969 raft_consensus.cc:348] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.233803 21969 raft_consensus.cc:374] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.233829 21969 raft_consensus.cc:729] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ed03f35e848b395e560fc47682e47, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.233886 21969 consensus_queue.cc:260] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.233999 21969 ts_tablet_manager.cc:1428] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:47.234297 21967 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: No bootstrap required, opened a new log
I20260925 10:02:47.234357 21967 ts_tablet_manager.cc:1397] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:47.234572 21967 raft_consensus.cc:348] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.234654 21967 raft_consensus.cc:374] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.234694 21967 raft_consensus.cc:729] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.234740 21967 consensus_queue.cc:260] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.234866 21967 ts_tablet_manager.cc:1428] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:02:47.235045 21968 tablet_bootstrap.cc:492] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: No bootstrap required, opened a new log
I20260925 10:02:47.235105 21968 ts_tablet_manager.cc:1397] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:02:47.235272 21968 raft_consensus.cc:348] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.235363 21968 raft_consensus.cc:374] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:47.235390 21968 raft_consensus.cc:729] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc, State: Initialized, Role: FOLLOWER
I20260925 10:02:47.235437 21968 consensus_queue.cc:260] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.235538 21968 ts_tablet_manager.cc:1428] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:47.577467 21978 raft_consensus.cc:482] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:47.577582 21978 raft_consensus.cc:504] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.577792 21978 leader_election.cc:288] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:47.578044 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2017d4715ce748698d04201e2ace4756" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47" is_pre_election: true
I20260925 10:02:47.578217 21666 raft_consensus.cc:2454] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 0.
I20260925 10:02:47.578205 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2017d4715ce748698d04201e2ace4756" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" is_pre_election: true
I20260925 10:02:47.578378 21526 raft_consensus.cc:2454] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 0.
I20260925 10:02:47.578435 21740 leader_election.cc:302] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed8e27680004bcdbf063f6a0324ce03, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:47.578522 21978 raft_consensus.cc:2781] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:47.578554 21978 raft_consensus.cc:482] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:47.578579 21978 raft_consensus.cc:3037] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.579386 21978 raft_consensus.cc:504] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.579543 21978 leader_election.cc:288] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785), dd1ed03f35e848b395e560fc47682e47 (127.19.46.194:35429)
I20260925 10:02:47.579689 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2017d4715ce748698d04201e2ace4756" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ed03f35e848b395e560fc47682e47"
I20260925 10:02:47.579785 21666 raft_consensus.cc:3037] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.579774 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2017d4715ce748698d04201e2ace4756" candidate_uuid: "4ed8e27680004bcdbf063f6a0324ce03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc"
I20260925 10:02:47.579895 21526 raft_consensus.cc:3037] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:47.580438 21666 raft_consensus.cc:2454] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 1.
I20260925 10:02:47.580615 21740 leader_election.cc:302] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed8e27680004bcdbf063f6a0324ce03, dd1ed03f35e848b395e560fc47682e47; no voters:
I20260925 10:02:47.580698 21526 raft_consensus.cc:2454] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed8e27680004bcdbf063f6a0324ce03 in term 1.
I20260925 10:02:47.580722 21978 raft_consensus.cc:2781] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:47.580766 21978 raft_consensus.cc:686] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed8e27680004bcdbf063f6a0324ce03, State: Running, Role: LEADER
I20260925 10:02:47.580822 21978 consensus_queue.cc:237] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } }
I20260925 10:02:47.581513 21367 catalog_manager.cc:5495] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "4ed8e27680004bcdbf063f6a0324ce03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed8e27680004bcdbf063f6a0324ce03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 38579 } health_report { overall_health: HEALTHY } } }
I20260925 10:02:47.593740 21526 raft_consensus.cc:1261] T 2017d4715ce748698d04201e2ace4756 P 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc [term 1 FOLLOWER]: Refusing update from remote peer 4ed8e27680004bcdbf063f6a0324ce03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:47.593909 21666 raft_consensus.cc:1261] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Refusing update from remote peer 4ed8e27680004bcdbf063f6a0324ce03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:02:47.594102 21987 consensus_queue.cc:1035] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4f3b5fe9549fab7dc5d5fa2d5e3fc" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:02:47.594254 21978 consensus_queue.cc:1035] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1ed03f35e848b395e560fc47682e47" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:02:47.598743 21938 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:02:49.116823 21367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:49.116942 21367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:02:49.117946 21367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f51075866e64d88bb06bcd9b2c8c8ae: Sending DeleteTablet for 3 replicas of tablet 2017d4715ce748698d04201e2ace4756
I20260925 10:02:49.118238 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2017d4715ce748698d04201e2ace4756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:49 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:33008
I20260925 10:02:49.118261 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet 2017d4715ce748698d04201e2ace4756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:49 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 10:02:49.118321 21786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2017d4715ce748698d04201e2ace4756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:02:49 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260925 10:02:49.118459 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 21441
I20260925 10:02:49.125560 22021 tablet_replica.cc:330] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: stopping tablet replica
I20260925 10:02:49.125679 22021 raft_consensus.cc:2229] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:02:49.125712 22022 tablet_replica.cc:330] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: stopping tablet replica
I20260925 10:02:49.125744 22021 raft_consensus.cc:2258] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:49.125808 22022 raft_consensus.cc:2229] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 LEADER]: Raft consensus shutting down.
W20260925 10:02:49.125797 21354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:42785 (error 108)
I20260925 10:02:49.125896 22022 raft_consensus.cc:2258] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:02:49.126076 22021 ts_tablet_manager.cc:1905] T 2017d4715ce748698d04201e2ace4756 P dd1ed03f35e848b395e560fc47682e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:49.126161 22022 ts_tablet_manager.cc:1905] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:02:49.126614 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 21581
I20260925 10:02:49.128356 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:49.126601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42785 (local address 127.0.0.1:49710)
0925 10:02:49.128269 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:49.128315 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":87,"threads_started":1}
W20260925 10:02:49.128489 21354 catalog_manager.cc:4584] TS 0ec4f3b5fe9549fab7dc5d5fa2d5e3fc (127.19.46.193:42785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2017d4715ce748698d04201e2ace4756: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42785: connect: Connection refused (error 111)
I20260925 10:02:49.128664 22022 ts_tablet_manager.cc:1918] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:02:49.128751 22022 log.cc:1196] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/2017d4715ce748698d04201e2ace4756
I20260925 10:02:49.129000 22022 ts_tablet_manager.cc:1939] T 2017d4715ce748698d04201e2ace4756 P 4ed8e27680004bcdbf063f6a0324ce03: Deleting consensus metadata
I20260925 10:02:49.129347 21354 catalog_manager.cc:4841] TS 4ed8e27680004bcdbf063f6a0324ce03 (127.19.46.195:38579): tablet 2017d4715ce748698d04201e2ace4756 (table test-table [id=9bcb512b0d6b4eca859011012794dea1]) successfully deleted
I20260925 10:02:49.132920 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 21721
I20260925 10:02:49.132951 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:49.132816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:35429 (local address 127.0.0.1:33040)
0925 10:02:49.132875 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:49.132909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.194:35429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:02:49.138041 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 21337
2026-09-25T10:02:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6133 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:49 dist-test-slave-f9rl krb5kdc[22027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:49 dist-test-slave-f9rl krb5kdc[22027](info): set up 2 sockets
Sep 25 10:02:49 dist-test-slave-f9rl krb5kdc[22027](info): commencing operation
krb5kdc: starting...
W20260925 10:02:50.234907 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:50.369272 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37211
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:50.451920 22040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:50.452205 22040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:50.452244 22040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:50.453604 22040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:50.453682 22040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:50.453706 22040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:50.453734 22040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:50.453780 22040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:50.455441 22040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:50.455744 22040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:50.456007 22040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:50.461246 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:50.461352 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.461408 22040 server_base.cc:1034] running on GCE node
W20260925 10:02:50.461256 22048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.461820 22040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:50.462146 22040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:50.463336 22040 hybrid_clock.cc:648] HybridClock initialized: now 1790330570463334 us; error 46 us; skew 500 ppm
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:50.466148 22040 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:50.466755 22040 webserver.cc:460] Webserver started at http://127.19.46.254:37109/ using document root <none> and password file <none>
I20260925 10:02:50.467049 22040 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:50.467128 22040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:50.467306 22040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:50.468544 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "19d6960bf3024679b50649b99ceadaaa"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "9240f4f15ea47cdfb55718086f13f157"
server_key_iv: "8d97f3e19b5eaa4c2ac047f6ba8090f5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.468971 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "19d6960bf3024679b50649b99ceadaaa"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "9240f4f15ea47cdfb55718086f13f157"
server_key_iv: "8d97f3e19b5eaa4c2ac047f6ba8090f5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.470597 22040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 10:02:50.471439 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.471748 22040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:50.471861 22040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "19d6960bf3024679b50649b99ceadaaa"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "9240f4f15ea47cdfb55718086f13f157"
server_key_iv: "8d97f3e19b5eaa4c2ac047f6ba8090f5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.471942 22040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:50.485946 22040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:50.488145 22040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:50.488338 22040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:50.493330 22040 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:37211
I20260925 10:02:50.493367 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:37211 every 8 connection(s)
I20260925 10:02:50.494009 22040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:50.494650 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22040
I20260925 10:02:50.494735 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:50.494871 22108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:50.494923 19643 external_mini_cluster.cc:1428] Setting key b86adedb748e56f59f7d32224539db7d
I20260925 10:02:50.498395 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa: Bootstrap starting.
I20260925 10:02:50.499145 22108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:50.499490 22108 log.cc:824] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:50.500337 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa: No bootstrap required, opened a new log
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:50.502637 22108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } }
I20260925 10:02:50.502838 22108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:50.502904 22108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d6960bf3024679b50649b99ceadaaa, State: Initialized, Role: FOLLOWER
I20260925 10:02:50.503074 22108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } }
I20260925 10:02:50.503173 22108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:50.503218 22108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:50.503268 22108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:50.503844 22108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } }
I20260925 10:02:50.503964 22108 leader_election.cc:302] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d6960bf3024679b50649b99ceadaaa; no voters:
I20260925 10:02:50.504159 22108 leader_election.cc:288] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:50.504213 22113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:50.504386 22113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [term 1 LEADER]: Becoming Leader. State: Replica: 19d6960bf3024679b50649b99ceadaaa, State: Running, Role: LEADER
I20260925 10:02:50.504468 22108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:50.504551 22113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } }
I20260925 10:02:50.505008 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d6960bf3024679b50649b99ceadaaa. Latest consensus state: current_term: 1 leader_uuid: "19d6960bf3024679b50649b99ceadaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } } }
I20260925 10:02:50.505173 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:50.505483 22122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:50.506074 22122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:50.506366 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d6960bf3024679b50649b99ceadaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d6960bf3024679b50649b99ceadaaa" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37211 } } }
I20260925 10:02:50.506587 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:50.507318 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:50.495492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.19.46.254:37211)
0925 10:02:50.495688 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:50.495695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:50.495722 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:02:50.496110 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:50.496116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:50.496163 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:50.496278 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:50.497137 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.498163 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.499270 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.499757 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.499850 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:50.502334 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:50.502339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:50.502339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:50.502353 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:50.504786 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.505373 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.505377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.505377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.505654 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.505930 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.505931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.505931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.506285 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:50.506448 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:50.506924 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 10:02:50.507105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260925 10:02:50.507889 22122 catalog_manager.cc:1347] Generated new cluster ID: 97ce24672bf243f185f99ca22686e2c4
I20260925 10:02:50.507931 22122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:02:50.521588 22122 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:50.522351 22122 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:50.528333 22122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d6960bf3024679b50649b99ceadaaa: Generated new TSK 0
I20260925 10:02:50.528560 22122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:50.561162 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:50.655419 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:50.655642 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:50.655668 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:50.656951 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:50.656998 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:50.657032 22134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:50.658666 22134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:50.658927 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:50.659188 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:50.665864 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:50.665867 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:50.665915 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.666294 22134 server_base.cc:1034] running on GCE node
I20260925 10:02:50.666579 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:50.666831 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:50.668018 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790330570668006 us; error 48 us; skew 500 ppm
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:50.671093 22134 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:50.671644 22134 webserver.cc:460] Webserver started at http://127.19.46.193:40745/ using document root <none> and password file <none>
I20260925 10:02:50.671907 22134 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:50.671984 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:50.672159 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:50.673408 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "61aadccf290145da8ae920e5d2b73512"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1a56342ffa6c07f9a0257a9ebecf28c3"
server_key_iv: "b165aa09a586ffc37f0b582729593bd5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.673905 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "61aadccf290145da8ae920e5d2b73512"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1a56342ffa6c07f9a0257a9ebecf28c3"
server_key_iv: "b165aa09a586ffc37f0b582729593bd5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.675468 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:50.676440 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.676819 22134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:50.676959 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "61aadccf290145da8ae920e5d2b73512"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1a56342ffa6c07f9a0257a9ebecf28c3"
server_key_iv: "b165aa09a586ffc37f0b582729593bd5"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.677058 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:50.685356 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:50.688001 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:50.688217 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:50.688544 22134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:50.688941 22134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:50.689007 22134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.689056 22134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:50.689085 22134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.695842 22134 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:46433
I20260925 10:02:50.695927 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:46433 every 8 connection(s)
I20260925 10:02:50.696298 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:50.697331 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22134
I20260925 10:02:50.697415 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:50.697674 19643 external_mini_cluster.cc:1428] Setting key 307c1e05d0462dd38a0f50b494e502e9
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:02:50.706542 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:50.697071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:60481 (local address 127.19.46.254:37211)
0925 10:02:50.697178 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:50.697181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:50.697688 (+ 507us) server_negotiation.cc:407] Connection header received
0925 10:02:50.698222 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:50.698223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:50.698234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:50.698291 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:50.699263 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.699778 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.700784 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.701019 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.701107 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:50.703379 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:50.703380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:50.703380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:50.703388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:50.704934 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.705371 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.705372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.705373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.705476 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.705815 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.705816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.705816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.705981 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:50.706064 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:50.706351 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:02:50.706441 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:02:50.706755 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:50.697033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37211 (local address 127.19.46.193:60481)
0925 10:02:50.697542 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:50.697592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:50.698111 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:50.698371 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:50.698375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:50.698953 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:50.699161 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:50.699165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.699899 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.699900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:50.700689 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:50.700691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.701126 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.701127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:50.701253 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:50.701882 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:50.701890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:50.703266 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:50.705027 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:50.705036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:50.705039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:50.705289 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:50.705532 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:50.705533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:50.705534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:50.705742 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:50.706053 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:50.706060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:50.706216 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 10:02:50.706332 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:02:50.706505 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":98,"threads_started":1}
I20260925 10:02:50.707387 22263 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37211
I20260925 10:02:50.707501 22263 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:50.707675 22263 heartbeater.cc:511] Master 127.19.46.254:37211 requested a full tablet report, sending...
I20260925 10:02:50.708380 22072 ts_manager.cc:194] Registered new tserver with Master: 61aadccf290145da8ae920e5d2b73512 (127.19.46.193:46433)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
I20260925 10:02:50.709221 22072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:60481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:50.725234 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:50.813613 22268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:50.813851 22268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:50.813879 22268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:50.815510 22268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:50.815645 22268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:50.815744 22268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:50.817420 22268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:50.817715 22268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:50.817962 22268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:50.823391 22276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.823545 22268 server_base.cc:1034] running on GCE node
W20260925 10:02:50.823421 22274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:50.823410 22273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.823858 22268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:50.824156 22268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:50.825381 22268 hybrid_clock.cc:648] HybridClock initialized: now 1790330570825353 us; error 27 us; skew 500 ppm
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:50.828994 22268 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:50.829969 22268 webserver.cc:460] Webserver started at http://127.19.46.194:36791/ using document root <none> and password file <none>
I20260925 10:02:50.830305 22268 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:50.830377 22268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:50.830538 22268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:50.831722 22268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "478c097132564aefb7e7f900347b8f91"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "2c4fe1df4ff35c1b1948619f7a8d3d27"
server_key_iv: "3a62fd961582ba44c4a8b26a52f46ca2"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.832113 22268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "478c097132564aefb7e7f900347b8f91"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "2c4fe1df4ff35c1b1948619f7a8d3d27"
server_key_iv: "3a62fd961582ba44c4a8b26a52f46ca2"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.833914 22268 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 10:02:50.835131 22283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.835479 22268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:50.835624 22268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "478c097132564aefb7e7f900347b8f91"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "2c4fe1df4ff35c1b1948619f7a8d3d27"
server_key_iv: "3a62fd961582ba44c4a8b26a52f46ca2"
server_key_version: "encryptionkey@0"
I20260925 10:02:50.835726 22268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:50.852243 22268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:50.854967 22268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:50.855150 22268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:50.855440 22268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:50.855778 22268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:50.855837 22268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.855881 22268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:50.855908 22268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:50.864436 22396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34569 every 8 connection(s)
I20260925 10:02:50.864429 22268 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34569
I20260925 10:02:50.865082 22268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:02:50.871305 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22268
I20260925 10:02:50.871439 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:50.871654 19643 external_mini_cluster.cc:1428] Setting key 0665cbf565d9763133624bb550a7170d
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:50.876830 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:50.866166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:35657 (local address 127.19.46.254:37211)
0925 10:02:50.866269 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:50.866272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:50.866449 (+ 177us) server_negotiation.cc:407] Connection header received
0925 10:02:50.867328 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:50.867331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:50.867493 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:50.867595 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:50.868710 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.869232 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.870385 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.870596 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.870686 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:50.873373 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:50.873375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:50.873375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:50.873401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:50.875156 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.875674 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.875675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.875676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.875744 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:50.876019 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:50.876020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:50.876020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:50.876164 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:50.876247 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:50.876625 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:02:50.876723 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:02:50.877063 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:50.866017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37211 (local address 127.19.46.194:35657)
0925 10:02:50.866300 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:50.866351 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:50.867102 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:50.867700 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:50.867712 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:50.868365 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:50.868590 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:50.868596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.869338 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.869338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:50.870268 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:50.870271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:50.870698 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:50.870699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:50.870836 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:50.871569 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:50.871578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:50.873257 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:50.875248 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:50.875255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:50.875258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:50.875560 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:50.875821 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:50.875822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:50.875822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:50.875944 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:50.876236 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:50.876243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:50.876414 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 10:02:50.876604 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 10:02:50.876801 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260925 10:02:50.877601 22397 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37211
I20260925 10:02:50.877696 22397 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:50.877918 22397 heartbeater.cc:511] Master 127.19.46.254:37211 requested a full tablet report, sending...
I20260925 10:02:50.878439 22072 ts_manager.cc:194] Registered new tserver with Master: 478c097132564aefb7e7f900347b8f91 (127.19.46.194:34569)
I20260925 10:02:50.878868 22072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:35657
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:50.901304 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:50.986855 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:50.987064 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:50.987090 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:50.988236 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:50.988274 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:50.988301 22402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:50.989858 22402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:50.990060 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:50.990240 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:50.995563 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:50.995612 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.995682 22402 server_base.cc:1034] running on GCE node
W20260925 10:02:50.995584 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:50.996002 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:50.996294 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:50.997480 22402 hybrid_clock.cc:648] HybridClock initialized: now 1790330570997452 us; error 41 us; skew 500 ppm
Sep 25 10:02:50 dist-test-slave-f9rl krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:51.000187 22402 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:51.000597 22402 webserver.cc:460] Webserver started at http://127.19.46.195:36579/ using document root <none> and password file <none>
I20260925 10:02:51.000833 22402 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:51.000880 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:51.001042 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:51.002204 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "75b8b53f722a47ee87478ad8f6b4aa53"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1c835bb07c9f3f8410f60a1ed21c8ef4"
server_key_iv: "5a0ade98e0cd8080b6f6d3c16c668df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:51.002568 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "75b8b53f722a47ee87478ad8f6b4aa53"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1c835bb07c9f3f8410f60a1ed21c8ef4"
server_key_iv: "5a0ade98e0cd8080b6f6d3c16c668df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:51.003983 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:51.004812 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:51.005005 22402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:51.005102 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "75b8b53f722a47ee87478ad8f6b4aa53"
format_stamp: "Formatted at 2026-09-25 10:02:50 on dist-test-slave-f9rl"
server_key: "1c835bb07c9f3f8410f60a1ed21c8ef4"
server_key_iv: "5a0ade98e0cd8080b6f6d3c16c668df6"
server_key_version: "encryptionkey@0"
I20260925 10:02:51.005219 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:51.015259 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:51.017714 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:51.017887 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:51.018149 22402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:51.018477 22402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:51.018535 22402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:51.018580 22402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:51.018604 22402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:51.024550 22402 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:34825
I20260925 10:02:51.024597 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:34825 every 8 connection(s)
I20260925 10:02:51.024925 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:51.027016 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22402
I20260925 10:02:51.027120 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:51.027305 19643 external_mini_cluster.cc:1428] Setting key 36a9719a56b515ae3adc2034f836a4de
Sep 25 10:02:51 dist-test-slave-f9rl krb5kdc[22027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330570, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:51.034732 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:51.025704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:35213 (local address 127.19.46.254:37211)
0925 10:02:51.025749 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:51.025751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:51.025949 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:02:51.026436 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:51.026437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:51.026446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:51.026508 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:51.027504 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:51.028348 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:51.029400 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:51.029675 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:51.029736 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:51.031907 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:51.031909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:51.031909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:51.031915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:51.033364 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:51.033792 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:51.033792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:51.033793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:51.033846 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:51.034127 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:51.034128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:51.034128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:51.034262 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:51.034314 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:51.034592 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:02:51.034661 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:02:51.035003 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:51.025653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37211 (local address 127.19.46.195:35213)
0925 10:02:51.025818 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:51.025861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:51.026352 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:51.026561 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:51.026564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:51.027160 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:51.027415 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:51.027421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:51.028483 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:51.028483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:51.029323 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:51.029325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:51.029754 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:51.029755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:51.029867 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:51.030364 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:51.030372 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:51.031814 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:51.033487 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:51.033495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:51.033498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:51.033727 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:51.033928 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:51.033929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:51.033929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:51.034057 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:51.034330 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:51.034336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:51.034462 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 10:02:51.034568 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:02:51.034701 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 10:02:51.035609 22531 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37211
I20260925 10:02:51.035735 22531 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:51.035931 22531 heartbeater.cc:511] Master 127.19.46.254:37211 requested a full tablet report, sending...
I20260925 10:02:51.036355 22072 ts_manager.cc:194] Registered new tserver with Master: 75b8b53f722a47ee87478ad8f6b4aa53 (127.19.46.195:34825)
I20260925 10:02:51.036774 22072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:35213
I20260925 10:02:51.039593 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:02:51.051141 22111 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:02:51.050257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.19.46.254:37211)
0925 10:02:51.050331 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:51.050332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:51.050441 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:02:51.050494 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:51.050495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:51.050505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:51.050566 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:51.051076 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49810: BlockingRecv error: recv got EOF from 127.0.0.1:49810 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:02:51.052121 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22134
I20260925 10:02:51.057111 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22268
I20260925 10:02:51.061618 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22402
I20260925 10:02:51.065816 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22040
2026-09-25T10:02:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1926 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:51 dist-test-slave-f9rl krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:51 dist-test-slave-f9rl krb5kdc[22543](info): set up 2 sockets
Sep 25 10:02:51 dist-test-slave-f9rl krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260925 10:02:52.123178 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:52.258097 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34697
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:52.341114 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:52.341434 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:52.341503 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:52.342764 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:52.342844 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:52.342868 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:52.342896 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:52.342926 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:52.344527 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:52.344836 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:52.345096 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:52.350675 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:52.350682 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:52.350772 22556 server_base.cc:1034] running on GCE node
W20260925 10:02:52.350696 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:52.351629 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:52.352226 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:52.353415 22556 hybrid_clock.cc:648] HybridClock initialized: now 1790330572353400 us; error 34 us; skew 500 ppm
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330572, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:52.365096 22556 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:52.365749 22556 webserver.cc:460] Webserver started at http://127.19.46.254:33369/ using document root <none> and password file <none>
I20260925 10:02:52.366010 22556 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:52.366089 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:52.366257 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:52.367452 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3"
format_stamp: "Formatted at 2026-09-25 10:02:52 on dist-test-slave-f9rl"
server_key: "05964d71b92d187ceda3800c8be3fdda"
server_key_iv: "561e2dd96271aa199dae4a3358cc7da4"
server_key_version: "encryptionkey@0"
I20260925 10:02:52.367887 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3"
format_stamp: "Formatted at 2026-09-25 10:02:52 on dist-test-slave-f9rl"
server_key: "05964d71b92d187ceda3800c8be3fdda"
server_key_iv: "561e2dd96271aa199dae4a3358cc7da4"
server_key_version: "encryptionkey@0"
I20260925 10:02:52.369567 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:52.370678 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:52.371174 22556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:52.371356 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3"
format_stamp: "Formatted at 2026-09-25 10:02:52 on dist-test-slave-f9rl"
server_key: "05964d71b92d187ceda3800c8be3fdda"
server_key_iv: "561e2dd96271aa199dae4a3358cc7da4"
server_key_version: "encryptionkey@0"
I20260925 10:02:52.371455 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:52.379175 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:52.381748 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:52.381924 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:52.386296 22556 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:34697
I20260925 10:02:52.386373 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:34697 every 8 connection(s)
I20260925 10:02:52.386910 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:52.387658 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:52.390378 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3: Bootstrap starting.
I20260925 10:02:52.391130 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:52.391465 22624 log.cc:824] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:52.392256 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3: No bootstrap required, opened a new log
I20260925 10:02:52.393685 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22556
I20260925 10:02:52.393779 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:52.393959 19643 external_mini_cluster.cc:1428] Setting key 2fbc675b93073256c789aa26a1c9d7f0
I20260925 10:02:52.394415 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } }
I20260925 10:02:52.394651 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:52.394734 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dd751d5f0d49ea92d2a22752e4c8f3, State: Initialized, Role: FOLLOWER
I20260925 10:02:52.394898 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } }
I20260925 10:02:52.394987 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:52.395038 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:52.395089 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:52.395856 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } }
I20260925 10:02:52.395982 22624 leader_election.cc:302] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4dd751d5f0d49ea92d2a22752e4c8f3; no voters:
I20260925 10:02:52.396193 22624 leader_election.cc:288] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:52.396219 22629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:52.396330 22629 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [term 1 LEADER]: Becoming Leader. State: Replica: e4dd751d5f0d49ea92d2a22752e4c8f3, State: Running, Role: LEADER
I20260925 10:02:52.396472 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:52.396485 22629 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } }
I20260925 10:02:52.396968 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4dd751d5f0d49ea92d2a22752e4c8f3. Latest consensus state: current_term: 1 leader_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } } }
I20260925 10:02:52.397058 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:52.397305 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dd751d5f0d49ea92d2a22752e4c8f3" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34697 } } }
I20260925 10:02:52.397428 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:52.397639 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:52.398221 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:52.399853 22636 catalog_manager.cc:1347] Generated new cluster ID: b0262edf15ac49889a068ef5d2e927c8
I20260925 10:02:52.399981 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:52.406590 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:52.394530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.19.46.254:34697)
0925 10:02:52.395195 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:52.395205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:52.395239 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:02:52.395367 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:52.395374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:52.395430 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:52.395599 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:52.396377 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:52.397828 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:52.398944 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:52.399576 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:52.399691 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:52.402655 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:52.402660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:52.402661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:52.402676 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:52.404753 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:52.405217 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:52.405220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:52.405221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:52.405354 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:52.405603 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:52.405604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:52.405604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:52.405936 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:52.406103 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:52.406224 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:02:52.406370 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":565,"threads_started":1}
I20260925 10:02:52.407563 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:02:52.414361 22636 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:52.415192 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:52.425158 22636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4dd751d5f0d49ea92d2a22752e4c8f3: Generated new TSK 0
I20260925 10:02:52.425472 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:02:52.428550 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:52.424054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.19.46.254:34697)
0925 10:02:52.424102 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:52.424104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:52.424228 (+ 124us) server_negotiation.cc:407] Connection header received
0925 10:02:52.424303 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:52.424304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:52.424316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:52.424399 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:52.424921 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:52.425743 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:52.426556 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:52.426894 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:52.426970 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:52.427884 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:52.427885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:52.427885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:02:52.428218 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:52.428247 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:02:52.428257 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:52.428341 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:52.428424 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:02:52.428458 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:02:52.432260 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22556
2026-09-25T10:02:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1364 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22657](info): set up 2 sockets
Sep 25 10:02:52 dist-test-slave-f9rl krb5kdc[22657](info): commencing operation
krb5kdc: starting...
W20260925 10:02:53.494683 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:53 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:53.630266 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45491
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:53.714721 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:53.714970 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:53.714996 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:53.716153 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:53.716192 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:53.716207 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:53.716220 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:53.716233 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:53.717695 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:53.717909 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:53.718099 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:53.723325 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:53.723328 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:53.723536 22670 server_base.cc:1034] running on GCE node
W20260925 10:02:53.723328 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:53.723845 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:53.724121 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:53.725296 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790330573725280 us; error 55 us; skew 500 ppm
Sep 25 10:02:53 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330573, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:53.728098 22670 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:53.728615 22670 webserver.cc:460] Webserver started at http://127.19.46.254:40443/ using document root <none> and password file <none>
I20260925 10:02:53.728878 22670 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:53.728948 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:53.729104 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:53.730279 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "8c332d63dd844d4ea1ad44b893ca1610"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "76a5c2532747a8469b27550d25db36b8"
server_key_iv: "fff32dc88c2027870635738d1bf16067"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.730662 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "8c332d63dd844d4ea1ad44b893ca1610"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "76a5c2532747a8469b27550d25db36b8"
server_key_iv: "fff32dc88c2027870635738d1bf16067"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.732116 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:53.732947 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:53.733264 22670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:53.733498 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "8c332d63dd844d4ea1ad44b893ca1610"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "76a5c2532747a8469b27550d25db36b8"
server_key_iv: "fff32dc88c2027870635738d1bf16067"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.733605 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:53.751662 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:53.754299 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:53.754515 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:53.759153 22670 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:45491
I20260925 10:02:53.759187 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:45491 every 8 connection(s)
I20260925 10:02:53.759588 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:53.760682 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:53.763299 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610: Bootstrap starting.
I20260925 10:02:53.763867 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:53.764137 22738 log.cc:824] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:53.764966 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610: No bootstrap required, opened a new log
I20260925 10:02:53.766736 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22670
I20260925 10:02:53.766851 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:53.766736 22738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } }
I20260925 10:02:53.766924 22738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:53.766974 22738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c332d63dd844d4ea1ad44b893ca1610, State: Initialized, Role: FOLLOWER
I20260925 10:02:53.767052 19643 external_mini_cluster.cc:1428] Setting key 5c8fe8790d6d826cb10d7f270ff11c92
I20260925 10:02:53.767149 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } }
I20260925 10:02:53.767239 22738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:53.767288 22738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:53.767340 22738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:53.768360 22738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } }
I20260925 10:02:53.768456 22738 leader_election.cc:302] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c332d63dd844d4ea1ad44b893ca1610; no voters:
I20260925 10:02:53.768673 22738 leader_election.cc:288] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:53.768731 22743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:53.768884 22743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [term 1 LEADER]: Becoming Leader. State: Replica: 8c332d63dd844d4ea1ad44b893ca1610, State: Running, Role: LEADER
I20260925 10:02:53.768959 22738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:53.769039 22743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } }
I20260925 10:02:53.769450 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c332d63dd844d4ea1ad44b893ca1610. Latest consensus state: current_term: 1 leader_uuid: "8c332d63dd844d4ea1ad44b893ca1610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } } }
I20260925 10:02:53.769474 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c332d63dd844d4ea1ad44b893ca1610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c332d63dd844d4ea1ad44b893ca1610" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45491 } } }
I20260925 10:02:53.769596 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:53.769788 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:53.769932 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:53.770468 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:53.772473 22752 catalog_manager.cc:1347] Generated new cluster ID: d93c2ae8f4c348cc94c61cb1edea225f
I20260925 10:02:53.772534 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:53 dist-test-slave-f9rl krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:53.779892 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:53.767563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.19.46.254:45491)
0925 10:02:53.767770 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:53.767782 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:53.767818 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:02:53.767943 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:53.767949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:53.767994 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:53.768187 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:53.769031 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:53.770535 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:53.771552 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:53.772208 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:53.772340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:53.775058 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:53.775062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:53.775063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:53.775076 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:53.777877 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:53.778294 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:53.778297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:53.778297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:53.778417 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:53.778613 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:53.778614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:53.778615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:53.778928 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:53.779440 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:53.779509 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:02:53.779654 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260925 10:02:53.806892 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:53.807677 22752 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:53.814514 22752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c332d63dd844d4ea1ad44b893ca1610: Generated new TSK 0
I20260925 10:02:53.814741 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:53.858841 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:53.960551 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:53.960789 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:53.960816 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:53.962167 22764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:53.962222 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:53.962252 22764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:53.963912 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:53.964207 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:53.964478 22764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:53.970314 22769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:53.970304 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:53.970317 22770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:53.970611 22764 server_base.cc:1034] running on GCE node
I20260925 10:02:53.970885 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:53.971228 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:53.972553 22764 hybrid_clock.cc:648] HybridClock initialized: now 1790330573972516 us; error 33 us; skew 500 ppm
Sep 25 10:02:53 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330573, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:53.975941 22764 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:53.976476 22764 webserver.cc:460] Webserver started at http://127.19.46.193:45525/ using document root <none> and password file <none>
I20260925 10:02:53.976756 22764 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:53.976810 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:53.976956 22764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:53.978330 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "a7c515bf2ab14924be9a302f49c1caf8"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "b9ab59acf0558a6f6a4fdd5adba5bbef"
server_key_iv: "f9a0d395e214fefaf13dc511bb84acec"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.978809 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "a7c515bf2ab14924be9a302f49c1caf8"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "b9ab59acf0558a6f6a4fdd5adba5bbef"
server_key_iv: "f9a0d395e214fefaf13dc511bb84acec"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.980382 22764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:53.981300 22779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:53.981611 22764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:53.981698 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "a7c515bf2ab14924be9a302f49c1caf8"
format_stamp: "Formatted at 2026-09-25 10:02:53 on dist-test-slave-f9rl"
server_key: "b9ab59acf0558a6f6a4fdd5adba5bbef"
server_key_iv: "f9a0d395e214fefaf13dc511bb84acec"
server_key_version: "encryptionkey@0"
I20260925 10:02:53.981805 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:54.001152 22764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:54.003834 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:54.004031 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:54.004303 22764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:54.004645 22764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:54.004703 22764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.004747 22764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:54.004776 22764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.011276 22764 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:41267
I20260925 10:02:54.011364 22892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:41267 every 8 connection(s)
I20260925 10:02:54.011713 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:54.016227 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22764
I20260925 10:02:54.016364 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:54.016575 19643 external_mini_cluster.cc:1428] Setting key 93817386da7fa0454065f770f18f91c5
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330573, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:54.022349 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.012514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:33461 (local address 127.19.46.254:45491)
0925 10:02:54.012561 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:54.012565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:54.012770 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:02:54.013294 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:54.013295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:54.013309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:54.013372 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:54.014381 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.015006 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.016090 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.016312 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.016422 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.019224 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:54.019225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:54.019226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:54.019235 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:54.020743 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.021214 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.021215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.021215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.021276 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.021633 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.021634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.021634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.021769 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:54.021848 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:54.022144 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:02:54.022244 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:02:54.022539 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.012483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45491 (local address 127.19.46.193:33461)
0925 10:02:54.012641 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:54.012686 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:54.013207 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:54.013453 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:54.013457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:54.014045 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:54.014253 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.014258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.015113 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.015113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.015934 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.015936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.016512 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.016514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.016637 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.017316 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:54.017326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:54.019118 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:54.020864 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.020872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.020874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.021129 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.021353 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.021354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.021355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.021531 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.021838 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:54.021845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:54.021995 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:02:54.022101 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:02:54.022284 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 10:02:54.023116 22893 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45491
I20260925 10:02:54.023213 22893 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:54.023411 22893 heartbeater.cc:511] Master 127.19.46.254:45491 requested a full tablet report, sending...
I20260925 10:02:54.023960 22702 ts_manager.cc:194] Registered new tserver with Master: a7c515bf2ab14924be9a302f49c1caf8 (127.19.46.193:41267)
I20260925 10:02:54.024596 22702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:33461
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:54.045032 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:54.128206 22898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:54.128453 22898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:54.128477 22898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:54.129756 22898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:54.129801 22898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:54.129849 22898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:54.131371 22898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:54.131639 22898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:54.131870 22898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:54.137019 22903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:54.137161 22906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:54.137204 22904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:54.137428 22898 server_base.cc:1034] running on GCE node
I20260925 10:02:54.137667 22898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:54.137883 22898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:54.139075 22898 hybrid_clock.cc:648] HybridClock initialized: now 1790330574139060 us; error 30 us; skew 500 ppm
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:54.141782 22898 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:54.142297 22898 webserver.cc:460] Webserver started at http://127.19.46.194:40629/ using document root <none> and password file <none>
I20260925 10:02:54.142553 22898 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:54.142623 22898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:54.142776 22898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:54.143857 22898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "e5c280c7451240fe9e2af382f5994840"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "59615b54f1ddf4f97544ee0e03fb9ac1"
server_key_iv: "00b634eb3f6edaee4760bc097098ac14"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.144228 22898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "e5c280c7451240fe9e2af382f5994840"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "59615b54f1ddf4f97544ee0e03fb9ac1"
server_key_iv: "00b634eb3f6edaee4760bc097098ac14"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.146006 22898 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:02:54.146911 22913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.147276 22898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:54.147403 22898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "e5c280c7451240fe9e2af382f5994840"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "59615b54f1ddf4f97544ee0e03fb9ac1"
server_key_iv: "00b634eb3f6edaee4760bc097098ac14"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.147497 22898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:54.166277 22898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:54.168673 22898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:54.168857 22898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:54.169168 22898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:54.169553 22898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:54.169606 22898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.169649 22898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:54.169677 22898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.175733 22898 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:43075
I20260925 10:02:54.175777 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:43075 every 8 connection(s)
I20260925 10:02:54.176105 22898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:02:54.180688 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 22898
I20260925 10:02:54.180824 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:54.181036 19643 external_mini_cluster.cc:1428] Setting key 734b717edbf7ded35f6ec42429d1b0eb
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:54.186475 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.176872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:33547 (local address 127.19.46.254:45491)
0925 10:02:54.176934 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:54.176936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:54.177186 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:02:54.177737 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:54.177738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:54.177749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:54.177808 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:54.178782 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.179309 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.180333 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.180534 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.180594 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.183229 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:54.183230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:54.183231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:54.183246 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:54.184780 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.185369 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.185370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.185371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.185490 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.185782 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.185783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.185783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.185909 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:54.185983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:54.186275 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:02:54.186371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:02:54.186690 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.176868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45491 (local address 127.19.46.194:33547)
0925 10:02:54.177041 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:54.177080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:54.177640 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:54.177882 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:54.177885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:54.178478 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:54.178684 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.178689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.179409 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.179409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.180245 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.180248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.180722 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.180723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.180891 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.181521 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:54.181531 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:54.183118 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:54.184955 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.184963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.184965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.185279 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.185565 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.185566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.185567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.185698 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.185975 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:54.185982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:54.186133 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 10:02:54.186255 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:02:54.186431 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260925 10:02:54.187198 23027 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45491
I20260925 10:02:54.187302 23027 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:54.187506 23027 heartbeater.cc:511] Master 127.19.46.254:45491 requested a full tablet report, sending...
I20260925 10:02:54.187970 22702 ts_manager.cc:194] Registered new tserver with Master: e5c280c7451240fe9e2af382f5994840 (127.19.46.194:43075)
I20260925 10:02:54.188524 22702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:33547
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:54.209100 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:54.294409 23032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:54.294623 23032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:54.294648 23032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:54.295763 23032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:54.295800 23032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:54.295826 23032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:54.297250 23032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:54.297524 23032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:54.297739 23032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:54.302886 23040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:54.302897 23038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:54.303189 23032 server_base.cc:1034] running on GCE node
W20260925 10:02:54.302881 23037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:54.303449 23032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:54.303742 23032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:54.304936 23032 hybrid_clock.cc:648] HybridClock initialized: now 1790330574304902 us; error 46 us; skew 500 ppm
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:54.307826 23032 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:54.308300 23032 webserver.cc:460] Webserver started at http://127.19.46.195:45401/ using document root <none> and password file <none>
I20260925 10:02:54.308555 23032 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:54.308631 23032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:54.308780 23032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:54.309945 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "2e2751d4f2fc46c1831596fc76031573"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "330b6cd4913db88da2e002a619d1a05e"
server_key_iv: "a7714b0bf776566174a0a9918289ec44"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.310319 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "2e2751d4f2fc46c1831596fc76031573"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "330b6cd4913db88da2e002a619d1a05e"
server_key_iv: "a7714b0bf776566174a0a9918289ec44"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.311721 23032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:54.312554 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.312762 23032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:54.312870 23032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "2e2751d4f2fc46c1831596fc76031573"
format_stamp: "Formatted at 2026-09-25 10:02:54 on dist-test-slave-f9rl"
server_key: "330b6cd4913db88da2e002a619d1a05e"
server_key_iv: "a7714b0bf776566174a0a9918289ec44"
server_key_version: "encryptionkey@0"
I20260925 10:02:54.312948 23032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:54.335304 23032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:54.337980 23032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:54.338168 23032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:54.338435 23032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:54.338797 23032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:54.338857 23032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.338902 23032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:54.338934 23032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:54.347266 23032 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:45263
I20260925 10:02:54.347412 23160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:45263 every 8 connection(s)
I20260925 10:02:54.347743 23032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:54.354792 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23032
I20260925 10:02:54.354875 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:54.355098 19643 external_mini_cluster.cc:1428] Setting key 192146febb1792a788ca288c33fb8a74
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:54.359995 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.348548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:51777 (local address 127.19.46.254:45491)
0925 10:02:54.348615 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:54.348617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:54.348867 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:02:54.349557 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:54.349559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:54.349585 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:54.349643 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:54.350936 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.351560 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.352815 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.353038 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.353121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.355944 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:54.355945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:54.355945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:54.355958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:54.358039 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.358607 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.358609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.358610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.358688 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.359132 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.359133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.359134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.359282 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:54.359351 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:54.359773 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 10:02:54.359898 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:02:54.360069 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.348523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45491 (local address 127.19.46.195:51777)
0925 10:02:54.348705 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:54.348749 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:54.349381 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:54.349711 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:54.349716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:54.350364 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:54.350755 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.350760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.351765 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.351766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.352686 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:54.352689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.353191 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.353193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:54.353326 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.353952 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:54.353961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:54.355792 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:54.358179 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.358189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.358191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.358479 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.358801 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:54.358802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:54.358802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:54.359003 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:54.359375 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:54.359382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:54.359536 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 10:02:54.359675 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:02:54.359858 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260925 10:02:54.360945 23161 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45491
I20260925 10:02:54.361312 23161 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:54.361707 23161 heartbeater.cc:511] Master 127.19.46.254:45491 requested a full tablet report, sending...
I20260925 10:02:54.362324 22702 ts_manager.cc:194] Registered new tserver with Master: 2e2751d4f2fc46c1831596fc76031573 (127.19.46.195:45263)
I20260925 10:02:54.362819 22702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:51777
I20260925 10:02:54.367733 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:54.393415 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:54.384781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.19.46.254:45491)
0925 10:02:54.384846 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:54.384848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:54.384900 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:02:54.384969 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:54.384970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:54.384979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:54.385043 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:54.385980 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.386600 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.387416 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:54.387634 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:54.387701 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:54.390763 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:54.390765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:54.390765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:54.390772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:54.392291 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.392660 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.392660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.392661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.392711 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:54.392923 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:54.392923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:54.392923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:54.393074 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:54.393185 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:54.393275 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 10:02:54.393341 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 10:02:54.393735 22702 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34768
I20260925 10:02:54.394189 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22764
I20260925 10:02:54.398965 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22898
I20260925 10:02:54.404129 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23032
I20260925 10:02:54.408615 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 22670
2026-09-25T10:02:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1978 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[23173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[23173](info): set up 2 sockets
Sep 25 10:02:54 dist-test-slave-f9rl krb5kdc[23173](info): commencing operation
krb5kdc: starting...
W20260925 10:02:55.559571 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 10:02:55.707086 19643 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.147s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 10:02:55.871507 19643 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.164s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 10:02:56.074905 19643 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.203s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/test-user.keytab.
W20260925 10:02:56.812391 19643 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.737s user 0.001s sys 0.001s
Sep 25 10:02:56 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
W20260925 10:02:57.001791 19643 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.19.46.254: real 0.167s user 0.000s sys 0.003s
I20260925 10:02:57.002188 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46611
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:57.097656 23186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:57.097900 23186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:57.097942 23186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:57.099238 23186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:57.099316 23186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:57.099344 23186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:57.099376 23186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:57.099402 23186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:57.101087 23186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:57.101402 23186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:57.101697 23186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:57.107613 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.107625 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.107625 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:57.107745 23186 server_base.cc:1034] running on GCE node
I20260925 10:02:57.108243 23186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:57.108558 23186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:57.109725 23186 hybrid_clock.cc:648] HybridClock initialized: now 1790330577109714 us; error 38 us; skew 500 ppm
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:57.112664 23186 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:57.113245 23186 webserver.cc:460] Webserver started at http://127.19.46.254:45477/ using document root <none> and password file <none>
I20260925 10:02:57.113595 23186 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:57.113651 23186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:57.113831 23186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:57.114970 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4e31a64e64254d989f2d27cbf77eb80c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "bb82d6cfb69fc3e6663ec7caf294e760"
server_key_iv: "924cb4e70d921c81c6075efbc00085ff"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.115350 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4e31a64e64254d989f2d27cbf77eb80c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "bb82d6cfb69fc3e6663ec7caf294e760"
server_key_iv: "924cb4e70d921c81c6075efbc00085ff"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.116861 23186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:02:57.118009 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.118482 23186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:57.118659 23186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4e31a64e64254d989f2d27cbf77eb80c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "bb82d6cfb69fc3e6663ec7caf294e760"
server_key_iv: "924cb4e70d921c81c6075efbc00085ff"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.118753 23186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:57.134382 23186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:57.136914 23186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:57.137132 23186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:57.141997 23186 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46611
I20260925 10:02:57.142087 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46611 every 8 connection(s)
I20260925 10:02:57.142685 23186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:57.143438 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:57.146096 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c: Bootstrap starting.
I20260925 10:02:57.146739 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:57.147060 23254 log.cc:824] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:57.147979 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c: No bootstrap required, opened a new log
I20260925 10:02:57.147977 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23186
I20260925 10:02:57.148096 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:57.148288 19643 external_mini_cluster.cc:1428] Setting key 91a8fce59cb5e9cc4c14ede0d8becd4a
I20260925 10:02:57.150497 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } }
I20260925 10:02:57.150684 23254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:57.150753 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e31a64e64254d989f2d27cbf77eb80c, State: Initialized, Role: FOLLOWER
I20260925 10:02:57.150900 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } }
I20260925 10:02:57.151001 23254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:57.151057 23254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:57.151109 23254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:57.151708 23254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } }
I20260925 10:02:57.151830 23254 leader_election.cc:302] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e31a64e64254d989f2d27cbf77eb80c; no voters:
I20260925 10:02:57.152040 23254 leader_election.cc:288] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:57.152067 23259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:57.152179 23259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [term 1 LEADER]: Becoming Leader. State: Replica: 4e31a64e64254d989f2d27cbf77eb80c, State: Running, Role: LEADER
I20260925 10:02:57.152328 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:57.152320 23259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } }
I20260925 10:02:57.152804 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e31a64e64254d989f2d27cbf77eb80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } } }
I20260925 10:02:57.152891 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:57.153349 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e31a64e64254d989f2d27cbf77eb80c. Latest consensus state: current_term: 1 leader_uuid: "4e31a64e64254d989f2d27cbf77eb80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e31a64e64254d989f2d27cbf77eb80c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46611 } } }
I20260925 10:02:57.153424 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:57.154685 23265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:57.156315 23265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:57.158345 23265 catalog_manager.cc:1347] Generated new cluster ID: c1323bdd37364ed4b05b06a907c84f05
I20260925 10:02:57.158396 23265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:02:57.162185 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.148857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.19.46.254:46611)
0925 10:02:57.149037 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.149046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.149072 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:02:57.150496 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.150505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.150577 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.150769 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.151562 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.152770 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.153852 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.154489 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.154641 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.157841 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.157846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.157846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.157860 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.160019 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.160447 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.160450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.160450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.160565 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.160796 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.160796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.160797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.161118 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.161752 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.161822 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.161961 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260925 10:02:57.164211 23265 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:57.164745 23265 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:57.170430 23265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e31a64e64254d989f2d27cbf77eb80c: Generated new TSK 0
I20260925 10:02:57.170656 23265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:57.197191 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:57.283802 23280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:57.284029 23280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:57.284055 23280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:57.285277 23280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:57.285322 23280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:57.285351 23280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:57.286947 23280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:57.287210 23280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:57.287467 23280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:57.292699 23288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.292713 23286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.292713 23285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:57.292797 23280 server_base.cc:1034] running on GCE node
I20260925 10:02:57.293182 23280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:57.293476 23280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:57.294659 23280 hybrid_clock.cc:648] HybridClock initialized: now 1790330577294636 us; error 62 us; skew 500 ppm
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:57.297686 23280 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:57.298152 23280 webserver.cc:460] Webserver started at http://127.19.46.193:35137/ using document root <none> and password file <none>
I20260925 10:02:57.298388 23280 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:57.298435 23280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:57.298580 23280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:57.299702 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "f9805e41d3e0489a9fb1736d79389335"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "e44b9e93aa6d53548c4bb352d507a971"
server_key_iv: "a7df2e9e9bf309835e915ff9e540531a"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.300065 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "f9805e41d3e0489a9fb1736d79389335"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "e44b9e93aa6d53548c4bb352d507a971"
server_key_iv: "a7df2e9e9bf309835e915ff9e540531a"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.301388 23280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:02:57.302196 23295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.302395 23280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:02:57.302493 23280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "f9805e41d3e0489a9fb1736d79389335"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "e44b9e93aa6d53548c4bb352d507a971"
server_key_iv: "a7df2e9e9bf309835e915ff9e540531a"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.302572 23280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:57.323572 23280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:57.326066 23280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:57.326221 23280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:57.326505 23280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:57.326834 23280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:57.326886 23280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.326924 23280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:57.326951 23280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.332861 23280 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:37187
I20260925 10:02:57.332911 23408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:37187 every 8 connection(s)
I20260925 10:02:57.333245 23280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:57.342991 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23280
I20260925 10:02:57.343137 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:57.343362 19643 external_mini_cluster.cc:1428] Setting key ce61b4b98047797ea6619978ff2d835b
I20260925 10:02:57.343622 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.334024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:45285 (local address 127.19.46.254:46611)
0925 10:02:57.334093 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.334096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.334304 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:02:57.335284 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.335285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.335301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.335356 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.336047 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.336565 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.337700 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.337917 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.337996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.340536 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.340538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.340538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.340554 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.342050 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.342519 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.342520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.342520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.342585 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.342850 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.342851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.342851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.342984 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.343063 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.343422 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.343518 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:02:57.343827 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.334014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46611 (local address 127.19.46.193:45285)
0925 10:02:57.334186 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:57.334222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:57.335166 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:57.335437 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:57.335441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:57.335717 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:57.335917 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.335921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.336671 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.336672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.337610 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.337613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.338005 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.338005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.338147 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.338805 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:57.338814 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:57.340424 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:57.342165 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.342176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.342179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.342428 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.342652 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.342654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.342654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.342781 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.343050 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:57.343065 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:57.343243 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 10:02:57.343388 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 10:02:57.343567 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260925 10:02:57.344326 23409 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46611
I20260925 10:02:57.344444 23409 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:57.344648 23409 heartbeater.cc:511] Master 127.19.46.254:46611 requested a full tablet report, sending...
I20260925 10:02:57.345192 23218 ts_manager.cc:194] Registered new tserver with Master: f9805e41d3e0489a9fb1736d79389335 (127.19.46.193:37187)
I20260925 10:02:57.345876 23218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:45285
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:57.373625 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:57.458756 23414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:57.458992 23414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:57.459023 23414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:57.460158 23414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:57.460202 23414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:57.460233 23414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:57.461807 23414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:57.462041 23414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:57.462277 23414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:02:57.467621 23414 server_base.cc:1034] running on GCE node
W20260925 10:02:57.467674 23422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.467728 23420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.467617 23419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:57.468073 23414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:57.468389 23414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:57.469522 23414 hybrid_clock.cc:648] HybridClock initialized: now 1790330577469491 us; error 45 us; skew 500 ppm
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:57.476779 23414 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:57.477371 23414 webserver.cc:460] Webserver started at http://127.19.46.194:42765/ using document root <none> and password file <none>
I20260925 10:02:57.477671 23414 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:57.477752 23414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:57.477918 23414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:57.479112 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "dba33759936c4eab8568a2301df21e8c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "02976279495cb578ac4eafc388dd1ab4"
server_key_iv: "5cf7d4ddcc743db851665a4a8ba17dbc"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.479516 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "dba33759936c4eab8568a2301df21e8c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "02976279495cb578ac4eafc388dd1ab4"
server_key_iv: "5cf7d4ddcc743db851665a4a8ba17dbc"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.481009 23414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:02:57.481956 23429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.482170 23414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:02:57.482290 23414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "dba33759936c4eab8568a2301df21e8c"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "02976279495cb578ac4eafc388dd1ab4"
server_key_iv: "5cf7d4ddcc743db851665a4a8ba17dbc"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.482384 23414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:57.510079 23414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:57.512454 23414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:57.512636 23414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:57.512884 23414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:57.513228 23414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:57.513284 23414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.513326 23414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:57.513355 23414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.519626 23414 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:45563
I20260925 10:02:57.519673 23542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:45563 every 8 connection(s)
I20260925 10:02:57.520008 23414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:57.529273 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23414
I20260925 10:02:57.529378 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:57.529665 19643 external_mini_cluster.cc:1428] Setting key 28bd485363769f52866485e9a2f7309e
I20260925 10:02:57.530788 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.520754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:44949 (local address 127.19.46.254:46611)
0925 10:02:57.520826 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.520828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.521108 (+ 280us) server_negotiation.cc:407] Connection header received
0925 10:02:57.522132 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.522133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.522143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.522209 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.522897 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.523598 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.524957 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.525170 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.525230 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.527527 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.527529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.527529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.527537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.529013 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.529544 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.529546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.529547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.529618 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.529938 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.529939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.529940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.530107 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.530171 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.530573 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.530673 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:02:57.530884 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.520734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46611 (local address 127.19.46.194:44949)
0925 10:02:57.520918 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:57.521013 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:57.522017 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:57.522264 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:57.522267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:57.522539 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:57.522792 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.522796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.523788 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.523789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.524862 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.524866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.525258 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.525260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.525370 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.525958 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:57.525967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:57.527399 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:57.529120 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.529127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.529128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.529415 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.529691 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.529692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.529692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.529836 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.530211 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:57.530218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:57.530384 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 10:02:57.530499 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:02:57.530677 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260925 10:02:57.531368 23543 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46611
I20260925 10:02:57.531481 23543 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:57.531778 23543 heartbeater.cc:511] Master 127.19.46.254:46611 requested a full tablet report, sending...
I20260925 10:02:57.532312 23218 ts_manager.cc:194] Registered new tserver with Master: dba33759936c4eab8568a2301df21e8c (127.19.46.194:45563)
I20260925 10:02:57.532919 23218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:44949
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:57.562427 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:57.650099 23548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:57.650321 23548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:57.650346 23548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:57.651474 23548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:57.651511 23548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:57.651538 23548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:57.653237 23548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:57.653635 23548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:57.653898 23548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:57.659339 23556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.659358 23553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:57.659345 23554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:57.659686 23548 server_base.cc:1034] running on GCE node
I20260925 10:02:57.659915 23548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:57.660193 23548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:57.661351 23548 hybrid_clock.cc:648] HybridClock initialized: now 1790330577661325 us; error 48 us; skew 500 ppm
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:57.664165 23548 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:57.664649 23548 webserver.cc:460] Webserver started at http://127.19.46.195:37115/ using document root <none> and password file <none>
I20260925 10:02:57.664909 23548 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:57.664985 23548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:57.665138 23548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:57.666296 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "f84521ce0cae46699769d0b2d7430eb3"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "848263c965b4990e7e07ed70f26ef428"
server_key_iv: "fa22fe6584d3b0bd13af921b53a0f1b9"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.666658 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "f84521ce0cae46699769d0b2d7430eb3"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "848263c965b4990e7e07ed70f26ef428"
server_key_iv: "fa22fe6584d3b0bd13af921b53a0f1b9"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.668160 23548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:02:57.668967 23563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.669207 23548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:57.669322 23548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "f84521ce0cae46699769d0b2d7430eb3"
format_stamp: "Formatted at 2026-09-25 10:02:57 on dist-test-slave-f9rl"
server_key: "848263c965b4990e7e07ed70f26ef428"
server_key_iv: "fa22fe6584d3b0bd13af921b53a0f1b9"
server_key_version: "encryptionkey@0"
I20260925 10:02:57.669402 23548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:57.684499 23548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:57.687044 23548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:57.687217 23548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:57.687472 23548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:57.687803 23548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:57.687857 23548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.687901 23548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:57.687927 23548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:57.693840 23548 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:45311
I20260925 10:02:57.693933 23676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:45311 every 8 connection(s)
I20260925 10:02:57.694239 23548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:02:57.698377 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23548
I20260925 10:02:57.698459 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:57.698662 19643 external_mini_cluster.cc:1428] Setting key aea849e34f9eb324542dc75ad844de02
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:57.703625 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.694963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:50077 (local address 127.19.46.254:46611)
0925 10:02:57.695049 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.695052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.695254 (+ 202us) server_negotiation.cc:407] Connection header received
0925 10:02:57.696213 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.696214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.696225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.696283 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.696899 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.697413 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.698544 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.698834 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.698905 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.701151 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.701152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.701152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.701158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.702426 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.702799 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.702800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.702800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.702851 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.703057 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.703057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.703058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.703162 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.703211 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.703475 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.703562 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:02:57.703785 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.694921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46611 (local address 127.19.46.195:50077)
0925 10:02:57.695130 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:57.695165 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:57.696085 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:57.696334 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:57.696337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:57.696605 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:57.696792 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.696796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.697602 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.697603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.698466 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:57.698469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.698929 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.698931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:57.699095 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.699692 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:57.699700 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:57.701044 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:57.702518 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.702524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.702526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.702733 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.702907 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:57.702907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:57.702908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:57.703002 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:57.703218 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:57.703222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:57.703337 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 10:02:57.703421 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:02:57.703554 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260925 10:02:57.704272 23677 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46611
I20260925 10:02:57.704375 23677 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:57.704555 23677 heartbeater.cc:511] Master 127.19.46.254:46611 requested a full tablet report, sending...
I20260925 10:02:57.704981 23218 ts_manager.cc:194] Registered new tserver with Master: f84521ce0cae46699769d0b2d7430eb3 (127.19.46.195:45311)
I20260925 10:02:57.705404 23218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:50077
I20260925 10:02:57.710337 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:02:57.734246 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.725978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.19.46.193:37187)
0925 10:02:57.726134 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.726140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.726153 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:02:57.726189 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.726191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.726205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.726272 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.726863 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.727699 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.728452 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.728797 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.728866 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.731317 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.731326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.731327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.731334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.732965 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.733337 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.733341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.733342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.733425 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.733682 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.733684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.733684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.733843 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.733995 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.734032 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.734098 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:57.743960 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:57.735620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.19.46.254:46611)
0925 10:02:57.735678 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:57.735680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:57.735690 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:57.735773 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:57.735774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:57.735785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:57.735877 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:57.736470 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.737179 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.738007 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:57.738187 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:57.738269 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:57.740840 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:57.740842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:57.740842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:57.740849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:57.742420 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.742769 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.742771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.742771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.742841 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:57.743149 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:57.743150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:57.743150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:57.743314 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:57.743413 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:57.743796 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 10:02:57.743890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 10:02:57.744367 23218 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578
I20260925 10:02:57.744832 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23280
I20260925 10:02:57.749480 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23414
I20260925 10:02:57.754014 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23548
I20260925 10:02:57.758555 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23186
2026-09-25T10:02:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3349 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23689](info): set up 2 sockets
Sep 25 10:02:57 dist-test-slave-f9rl krb5kdc[23689](info): commencing operation
krb5kdc: starting...
W20260925 10:02:58.807067 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:02:58 dist-test-slave-f9rl krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:02:58.937045 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40077
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:59.018702 23702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:59.018918 23702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:59.018941 23702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:59.020032 23702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:02:59.020069 23702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:59.020083 23702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:59.020097 23702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:02:59.020108 23702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:02:59.021724 23702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:59.021929 23702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:59.022104 23702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:59.026955 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.026965 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.027031 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:59.027237 23702 server_base.cc:1034] running on GCE node
I20260925 10:02:59.027550 23702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:59.027869 23702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:59.029062 23702 hybrid_clock.cc:648] HybridClock initialized: now 1790330579029025 us; error 57 us; skew 500 ppm
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:59.031818 23702 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:02:59.032223 23702 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:59.032303 23702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:59.032485 23702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:59.033672 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "ea60eb7954e24353a4935be3e0589284"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "073d35fe2f49ac34ec750d8d3a791def"
server_key_iv: "1af13d314d2e3c8346c803ea8bfff389"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.034045 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "ea60eb7954e24353a4935be3e0589284"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "073d35fe2f49ac34ec750d8d3a791def"
server_key_iv: "1af13d314d2e3c8346c803ea8bfff389"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.035444 23702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:59.036392 23716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.036713 23702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:59.036871 23702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "ea60eb7954e24353a4935be3e0589284"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "073d35fe2f49ac34ec750d8d3a791def"
server_key_iv: "1af13d314d2e3c8346c803ea8bfff389"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.036973 23702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:59.056839 23702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:59.059370 23702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:59.059562 23702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:59.063805 23702 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:40077
I20260925 10:02:59.063838 23768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:40077 every 8 connection(s)
I20260925 10:02:59.064154 23702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:02:59.065160 23769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:59.067629 23769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: Bootstrap starting.
I20260925 10:02:59.068257 23769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:59.068562 23769 log.cc:824] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:59.069270 23769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: No bootstrap required, opened a new log
I20260925 10:02:59.070904 23769 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } }
I20260925 10:02:59.071046 23769 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:59.071072 23769 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea60eb7954e24353a4935be3e0589284, State: Initialized, Role: FOLLOWER
I20260925 10:02:59.071228 23769 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } }
I20260925 10:02:59.071305 23769 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:02:59.071327 23769 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:02:59.071355 23769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:59.072271 23769 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } }
I20260925 10:02:59.072402 23769 leader_election.cc:302] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea60eb7954e24353a4935be3e0589284; no voters:
I20260925 10:02:59.072451 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23702
I20260925 10:02:59.072569 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:02:59.072598 23769 leader_election.cc:288] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:02:59.072679 23772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:59.072786 19643 external_mini_cluster.cc:1428] Setting key 2d171fd40563861ec65f27a7105337c5
I20260925 10:02:59.072868 23769 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:02:59.072867 23772 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [term 1 LEADER]: Becoming Leader. State: Replica: ea60eb7954e24353a4935be3e0589284, State: Running, Role: LEADER
I20260925 10:02:59.073169 23772 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } }
I20260925 10:02:59.073621 23773 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea60eb7954e24353a4935be3e0589284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } } }
I20260925 10:02:59.073679 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea60eb7954e24353a4935be3e0589284. Latest consensus state: current_term: 1 leader_uuid: "ea60eb7954e24353a4935be3e0589284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea60eb7954e24353a4935be3e0589284" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40077 } } }
I20260925 10:02:59.073735 23773 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:59.073738 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284 [sys.catalog]: This master's current role is: LEADER
I20260925 10:02:59.074173 23783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:02:59.074970 23783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:02:59.076563 23783 catalog_manager.cc:1347] Generated new cluster ID: 4e9dacb7e90c43c096cdaf73962825fa
I20260925 10:02:59.076622 23783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:59.085235 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.073390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.19.46.254:40077)
0925 10:02:59.073602 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.073612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.073807 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:02:59.073945 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.073952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.074011 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.074172 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:59.075143 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.076485 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.077605 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.078231 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.078334 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.081306 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:59.081312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:59.081312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:59.081329 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:59.083458 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.083872 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.083876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.083876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.084003 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.084218 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.084219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.084219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.084565 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:59.084732 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.084828 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.084985 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":137,"threads_started":1}
I20260925 10:02:59.087216 23783 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:02:59.088085 23783 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:02:59.093132 23783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: Generated new TSK 0
I20260925 10:02:59.093394 23783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:02:59.126710 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:59.216219 23795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:59.216485 23795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:59.216527 23795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:59.217785 23795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:59.217864 23795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:59.217913 23795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:02:59.219614 23795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:59.219911 23795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:59.220170 23795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:59.226218 23800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.226235 23803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.226188 23801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:59.226935 23795 server_base.cc:1034] running on GCE node
I20260925 10:02:59.227205 23795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:59.227537 23795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:59.228737 23795 hybrid_clock.cc:648] HybridClock initialized: now 1790330579228719 us; error 31 us; skew 500 ppm
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:59.231779 23795 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:02:59.232246 23795 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:59.232324 23795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:59.232481 23795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:59.233731 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "f0360707e3684c2aa75a7c82f330fce9"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "45c9944b4ead1808dcd1f401db8dc763"
server_key_iv: "6fdb6c031fa00920608a72b5a4fc7928"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.234110 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "f0360707e3684c2aa75a7c82f330fce9"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "45c9944b4ead1808dcd1f401db8dc763"
server_key_iv: "6fdb6c031fa00920608a72b5a4fc7928"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.235533 23795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:59.236433 23809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.236666 23795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:59.236773 23795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "f0360707e3684c2aa75a7c82f330fce9"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "45c9944b4ead1808dcd1f401db8dc763"
server_key_iv: "6fdb6c031fa00920608a72b5a4fc7928"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.236862 23795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:59.247725 23795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:59.250056 23795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:59.250232 23795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:59.250494 23795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:59.250818 23795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:59.250873 23795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.250908 23795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:59.250957 23795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.257278 23795 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:33395
I20260925 10:02:59.257364 23922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:33395 every 8 connection(s)
I20260925 10:02:59.257714 23795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:02:59.262706 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23795
I20260925 10:02:59.262818 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:02:59.263034 19643 external_mini_cluster.cc:1428] Setting key 6fe3be6164873222f6fbde2bf1a7ed49
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:59.268426 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.258506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:47689 (local address 127.19.46.254:40077)
0925 10:02:59.258694 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.258702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.258774 (+ 72us) server_negotiation.cc:407] Connection header received
0925 10:02:59.259309 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.259311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.259322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.259393 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:59.260332 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.260853 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.262054 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.262256 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.262328 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.264822 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:59.264823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:59.264824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:59.264832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:59.266692 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.267153 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.267155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.267155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.267220 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.267517 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.267518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.267519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.267697 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:59.267777 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.268182 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.268277 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:02:59.268441 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.258457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40077 (local address 127.19.46.193:47689)
0925 10:02:59.258627 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.258669 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.259172 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.259447 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.259450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.259719 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:59.260225 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.260230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.260967 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.260968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.261950 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.261954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.262347 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.262348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.262458 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.263163 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:59.263173 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:59.264720 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:59.266793 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.266800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.266802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.267071 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.267324 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.267325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.267325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.267438 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.267879 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:59.267886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:59.268033 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.268129 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.268264 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260925 10:02:59.269173 23923 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40077
I20260925 10:02:59.269269 23923 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:59.269482 23923 heartbeater.cc:511] Master 127.19.46.254:40077 requested a full tablet report, sending...
I20260925 10:02:59.270073 23733 ts_manager.cc:194] Registered new tserver with Master: f0360707e3684c2aa75a7c82f330fce9 (127.19.46.193:33395)
I20260925 10:02:59.270741 23733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:47689
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:02:59.294092 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:59.385927 23928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:59.386214 23928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:59.386255 23928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:59.387501 23928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:59.387603 23928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:59.387657 23928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:02:59.389331 23928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:59.389667 23928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:59.389925 23928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:59.395371 23936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:59.395462 23928 server_base.cc:1034] running on GCE node
W20260925 10:02:59.395382 23933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.395382 23934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:59.395804 23928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:59.396075 23928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:59.397248 23928 hybrid_clock.cc:648] HybridClock initialized: now 1790330579397227 us; error 37 us; skew 500 ppm
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:59.400233 23928 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:02:59.400657 23928 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:59.400712 23928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:59.400811 23928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:59.402025 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "a6ba2d18b04546ec88e723b82c987e54"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "9d78f75dfffb850cd5d1d8d03f2ae2f2"
server_key_iv: "7753c1127ebcf001385eea766176c22b"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.402370 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "a6ba2d18b04546ec88e723b82c987e54"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "9d78f75dfffb850cd5d1d8d03f2ae2f2"
server_key_iv: "7753c1127ebcf001385eea766176c22b"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.403743 23928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:59.404742 23942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.405045 23928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:59.405164 23928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "a6ba2d18b04546ec88e723b82c987e54"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "9d78f75dfffb850cd5d1d8d03f2ae2f2"
server_key_iv: "7753c1127ebcf001385eea766176c22b"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.405267 23928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:59.421811 23928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:59.424263 23928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:59.424386 23928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:59.424588 23928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:59.424933 23928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:59.424973 23928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.424997 23928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:59.425009 23928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.431691 23928 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37805
I20260925 10:02:59.431769 24055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37805 every 8 connection(s)
I20260925 10:02:59.432085 23928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:59.441263 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 23928
I20260925 10:02:59.441459 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:02:59.441694 19643 external_mini_cluster.cc:1428] Setting key b752dd77d5d1af26fffbf2fa1500c8d8
I20260925 10:02:59.443302 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.432912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:50495 (local address 127.19.46.254:40077)
0925 10:02:59.432980 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.432982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.433218 (+ 236us) server_negotiation.cc:407] Connection header received
0925 10:02:59.433808 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.433809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.433820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.433877 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:59.434842 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.435398 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.436473 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.436666 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.436735 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.439658 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:59.439660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:59.439660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:59.439670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:59.441363 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.442018 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.442020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.442020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.442138 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.442483 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.442485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.442485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.442651 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:59.442732 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.443048 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.443153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:02:59.443872 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.432884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40077 (local address 127.19.46.194:50495)
0925 10:02:59.433091 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.433125 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.433708 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.433947 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.433950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.434225 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:59.434726 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.434730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.435514 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.435515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.436372 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.436375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.436752 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.436753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.436859 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.437581 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:59.437590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:59.439522 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:59.441497 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.441508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.441511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.441858 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.442254 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.442254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.442255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.442399 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.442753 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:59.442760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:59.442897 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.443000 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.443368 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"spinlock_wait_cycles":9600,"thread_start_us":102,"threads_started":1}
I20260925 10:02:59.444455 24056 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40077
I20260925 10:02:59.444602 24056 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:59.444777 24056 heartbeater.cc:511] Master 127.19.46.254:40077 requested a full tablet report, sending...
I20260925 10:02:59.445271 23733 ts_manager.cc:194] Registered new tserver with Master: a6ba2d18b04546ec88e723b82c987e54 (127.19.46.194:37805)
I20260925 10:02:59.445874 23733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:50495
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:02:59.475268 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:02:59.564481 24061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:02:59.564709 24061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:02:59.564735 24061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:02:59.566022 24061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:02:59.566073 24061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:02:59.566103 24061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:02:59.567725 24061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:02:59.568005 24061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:02:59.568282 24061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:02:59.573957 24069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.573961 24066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:02:59.573997 24067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:02:59.574724 24061 server_base.cc:1034] running on GCE node
I20260925 10:02:59.574994 24061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:02:59.575289 24061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:02:59.576483 24061 hybrid_clock.cc:648] HybridClock initialized: now 1790330579576470 us; error 34 us; skew 500 ppm
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:02:59.579994 24061 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:02:59.580498 24061 fs_manager.cc:359] Metadata directory not provided
I20260925 10:02:59.580577 24061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:02:59.580717 24061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:02:59.582069 24061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "7386dd6da6394dc4bf32ce245fa9e565"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "47fa44087ba17e56adc8d9ffd85a5446"
server_key_iv: "fa362f628f46eb083fc3dbb13b0fc394"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.582448 24061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "7386dd6da6394dc4bf32ce245fa9e565"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "47fa44087ba17e56adc8d9ffd85a5446"
server_key_iv: "fa362f628f46eb083fc3dbb13b0fc394"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.583827 24061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:02:59.584647 24075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.584862 24061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:02:59.584969 24061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "7386dd6da6394dc4bf32ce245fa9e565"
format_stamp: "Formatted at 2026-09-25 10:02:59 on dist-test-slave-f9rl"
server_key: "47fa44087ba17e56adc8d9ffd85a5446"
server_key_iv: "fa362f628f46eb083fc3dbb13b0fc394"
server_key_version: "encryptionkey@0"
I20260925 10:02:59.585063 24061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:02:59.603211 24061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:02:59.605545 24061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:02:59.605737 24061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:02:59.605993 24061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:02:59.606324 24061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:02:59.606383 24061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.606431 24061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:02:59.606463 24061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:02:59.613089 24061 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:35359
I20260925 10:02:59.613158 24188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:35359 every 8 connection(s)
I20260925 10:02:59.613479 24061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:02:59 dist-test-slave-f9rl krb5kdc[23689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330579, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:02:59.621629 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24061
I20260925 10:02:59.621737 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:02:59.621953 19643 external_mini_cluster.cc:1428] Setting key 6dd06e22518b547c87e2f3d5f2707e6c
I20260925 10:02:59.624408 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.614280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:34357 (local address 127.19.46.254:40077)
0925 10:02:59.614379 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.614382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.614506 (+ 124us) server_negotiation.cc:407] Connection header received
0925 10:02:59.615018 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.615020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.615032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.615093 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:59.616072 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.616610 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.617750 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.617976 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.618041 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.621091 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:59.621093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:59.621093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:59.621100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:59.622916 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.623379 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.623380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.623380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.623443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.623705 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.623706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.623706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.623839 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:59.623941 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.624224 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.624308 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:02:59.624653 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.614188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40077 (local address 127.19.46.195:34357)
0925 10:02:59.614361 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.614411 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.614887 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.615146 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.615149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.615442 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:02:59.615960 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.615966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.616721 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.616722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.617667 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.617670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.618245 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.618252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.618497 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.619374 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:02:59.619385 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:02:59.620969 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:02:59.623028 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.623036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.623039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.623282 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.623520 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:02:59.623521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:02:59.623521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:02:59.623636 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:02:59.623907 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:02:59.623913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:02:59.624086 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.624201 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.624375 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260925 10:02:59.625173 24189 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40077
I20260925 10:02:59.625263 24189 heartbeater.cc:464] Registering TS with master...
I20260925 10:02:59.625383 24189 heartbeater.cc:511] Master 127.19.46.254:40077 requested a full tablet report, sending...
I20260925 10:02:59.625824 23733 ts_manager.cc:194] Registered new tserver with Master: 7386dd6da6394dc4bf32ce245fa9e565 (127.19.46.195:35359)
I20260925 10:02:59.626225 23733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:34357
I20260925 10:02:59.633872 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:02:59.641620 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.635352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.19.46.254:40077)
0925 10:02:59.635428 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.635430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.635491 (+ 61us) server_negotiation.cc:407] Connection header received
0925 10:02:59.635573 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.635574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.635585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.635635 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:02:59.636266 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.636770 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.637546 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.637749 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.637827 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.638984 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:02:59.638985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:02:59.638986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:02:59.638993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:02:59.640433 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.640875 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.640876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.640877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.640957 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:02:59.641108 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:02:59.641108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:02:59.641109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:02:59.641258 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:02:59.641355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.641418 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.641549 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:02:59.643208 23733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:02:59.644788 23733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:02:59.651597 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.19.46.193:33395)
0925 10:02:59.647828 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.647832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.647842 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:59.647879 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.647881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.647899 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.648040 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:59.648288 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.649077 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.650644 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.651131 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.651234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.651299 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.651451 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.651495 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":90,"threads_started":1}
I20260925 10:02:59.651587 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33395 (local address 127.0.0.1:50098)
0925 10:02:59.647427 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.647461 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.647531 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.648038 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.648039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.648042 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:59.648190 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.648193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.649211 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.649212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.650551 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.650554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.651238 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.651240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.651323 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.651337 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.651433 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.651495 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":44,"thread_start_us":100,"threads_started":1}
I20260925 10:02:59.651684 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.19.46.195:35359)
0925 10:02:59.648166 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.648170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.648178 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:02:59.648214 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.648217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.648232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.648306 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:59.648446 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.649121 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.650455 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.651203 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.651272 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.651321 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.651540 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.651579 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":76,"threads_started":1}
I20260925 10:02:59.651886 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:35359 (local address 127.0.0.1:59712)
0925 10:02:59.647704 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.647734 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.647783 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.648354 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.648354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.648356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:59.648392 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.648394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.649224 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.649226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.650360 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.650363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.651353 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.651355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.651412 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.651416 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.651458 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.651501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":79,"thread_start_us":78,"threads_started":1}
I20260925 10:02:59.652554 23857 tablet_service.cc:1467] Processing CreateTablet for tablet 527dc6254df341948479cc8ad0f611be (DEFAULT_TABLE table=test-table [id=ff6978ef26f5474b925e281752f51b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:59.652568 24123 tablet_service.cc:1467] Processing CreateTablet for tablet 527dc6254df341948479cc8ad0f611be (DEFAULT_TABLE table=test-table [id=ff6978ef26f5474b925e281752f51b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:59.652949 24123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527dc6254df341948479cc8ad0f611be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:59.652952 23857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527dc6254df341948479cc8ad0f611be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:59.652985 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37805 (local address 127.0.0.1:42450)
0925 10:02:59.647417 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.647462 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.647536 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.648003 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.648006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.648009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:59.648092 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.648095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.650524 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.650525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.651566 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.651568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.652715 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.652716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.652793 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.652797 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.652864 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.652910 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260925 10:02:59.653028 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.647617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.19.46.194:37805)
0925 10:02:59.647804 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.647810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.647821 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:02:59.647860 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.647863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.647887 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.648020 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:59.649320 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.650429 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.651653 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.652615 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.652694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.652747 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.652895 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.652941 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 10:02:59.654129 23990 tablet_service.cc:1467] Processing CreateTablet for tablet 527dc6254df341948479cc8ad0f611be (DEFAULT_TABLE table=test-table [id=ff6978ef26f5474b925e281752f51b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:02:59.654520 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527dc6254df341948479cc8ad0f611be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:02:59.655942 24206 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Bootstrap starting.
I20260925 10:02:59.656433 24207 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Bootstrap starting.
I20260925 10:02:59.656644 24206 tablet_bootstrap.cc:654] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:59.657186 24207 tablet_bootstrap.cc:654] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:59.657358 24206 log.cc:824] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:59.657917 24207 log.cc:824] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:59.658077 24209 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Bootstrap starting.
I20260925 10:02:59.658263 24206 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: No bootstrap required, opened a new log
I20260925 10:02:59.658346 24206 ts_tablet_manager.cc:1397] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:02:59.658756 24207 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: No bootstrap required, opened a new log
I20260925 10:02:59.658830 24207 ts_tablet_manager.cc:1397] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:02:59.658849 24209 tablet_bootstrap.cc:654] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Neither blocks nor log segments found. Creating new log.
I20260925 10:02:59.659463 24209 log.cc:824] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Log is configured to *not* fsync() on all Append() calls
I20260925 10:02:59.659940 24206 raft_consensus.cc:348] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.660131 24206 raft_consensus.cc:374] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:59.660174 24206 raft_consensus.cc:729] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7386dd6da6394dc4bf32ce245fa9e565, State: Initialized, Role: FOLLOWER
I20260925 10:02:59.660303 24206 consensus_queue.cc:260] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.660445 24209 tablet_bootstrap.cc:492] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: No bootstrap required, opened a new log
I20260925 10:02:59.660506 24209 ts_tablet_manager.cc:1397] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:02:59.660599 24206 ts_tablet_manager.cc:1428] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:02:59.660611 24189 heartbeater.cc:503] Master 127.19.46.254:40077 was elected leader, sending a full tablet report...
I20260925 10:02:59.661075 24207 raft_consensus.cc:348] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.661223 24207 raft_consensus.cc:374] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:59.661257 24207 raft_consensus.cc:729] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0360707e3684c2aa75a7c82f330fce9, State: Initialized, Role: FOLLOWER
I20260925 10:02:59.661362 24207 consensus_queue.cc:260] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.661679 24207 ts_tablet_manager.cc:1428] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:02:59.661759 23923 heartbeater.cc:503] Master 127.19.46.254:40077 was elected leader, sending a full tablet report...
I20260925 10:02:59.662456 24209 raft_consensus.cc:348] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.662573 24209 raft_consensus.cc:374] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:02:59.662593 24209 raft_consensus.cc:729] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ba2d18b04546ec88e723b82c987e54, State: Initialized, Role: FOLLOWER
I20260925 10:02:59.662722 24209 consensus_queue.cc:260] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.662992 24209 ts_tablet_manager.cc:1428] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:02:59.663053 24056 heartbeater.cc:503] Master 127.19.46.254:40077 was elected leader, sending a full tablet report...
W20260925 10:02:59.682680 24057 tablet.cc:2369] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:02:59.758553 23924 tablet.cc:2369] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:02:59.864039 24190 tablet.cc:2369] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:02:59.932916 24214 raft_consensus.cc:482] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:02:59.933100 24214 raft_consensus.cc:504] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.933555 24214 leader_election.cc:288] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0360707e3684c2aa75a7c82f330fce9 (127.19.46.193:33395), 7386dd6da6394dc4bf32ce245fa9e565 (127.19.46.195:35359)
I20260925 10:02:59.936872 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.933912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57605 (local address 127.19.46.193:33395)
0925 10:02:59.933973 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.933975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.933989 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:02:59.934085 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.934086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.934095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.934150 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:59.934338 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.934926 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.935994 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.936533 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.936599 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.936608 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.936754 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.936792 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:02:59.936895 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.934093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:55107 (local address 127.19.46.195:35359)
0925 10:02:59.934164 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:02:59.934167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:02:59.934177 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:02:59.934212 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:02:59.934213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:02:59.934222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:02:59.934299 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:02:59.934491 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.935015 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.935963 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.936543 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.936607 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.936615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:02:59.936779 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:02:59.936815 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:02:59.936911 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.933824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33395 (local address 127.19.46.194:57605)
0925 10:02:59.933920 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.933934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.933990 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.934207 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.934208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.934210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:59.934253 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.934254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.935079 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.935080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.935856 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.935858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.936594 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.936595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.936674 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.936678 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.936748 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.936791 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260925 10:02:59.936919 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:02:59.933953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:35359 (local address 127.19.46.194:55107)
0925 10:02:59.934093 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:02:59.934100 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 10:02:59.934165 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:02:59.934301 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:02:59.934302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:02:59.934304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:02:59.934399 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.934400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.935098 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.935099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.935866 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:02:59.935868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:02:59.936617 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:02:59.936618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:02:59.936701 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:02:59.936704 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:02:59.936773 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:02:59.936815 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260925 10:02:59.937233 23877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527dc6254df341948479cc8ad0f611be" candidate_uuid: "a6ba2d18b04546ec88e723b82c987e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0360707e3684c2aa75a7c82f330fce9" is_pre_election: true
I20260925 10:02:59.937378 23877 raft_consensus.cc:2454] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ba2d18b04546ec88e723b82c987e54 in term 0.
I20260925 10:02:59.937336 24143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527dc6254df341948479cc8ad0f611be" candidate_uuid: "a6ba2d18b04546ec88e723b82c987e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7386dd6da6394dc4bf32ce245fa9e565" is_pre_election: true
I20260925 10:02:59.937521 24143 raft_consensus.cc:2454] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ba2d18b04546ec88e723b82c987e54 in term 0.
I20260925 10:02:59.937628 23943 leader_election.cc:302] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6ba2d18b04546ec88e723b82c987e54, f0360707e3684c2aa75a7c82f330fce9; no voters:
I20260925 10:02:59.937930 24214 raft_consensus.cc:2781] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:02:59.937979 24214 raft_consensus.cc:482] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:02:59.937995 24214 raft_consensus.cc:3037] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:59.938699 24214 raft_consensus.cc:504] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.938838 24214 leader_election.cc:288] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [CANDIDATE]: Term 1 election: Requested vote from peers f0360707e3684c2aa75a7c82f330fce9 (127.19.46.193:33395), 7386dd6da6394dc4bf32ce245fa9e565 (127.19.46.195:35359)
I20260925 10:02:59.939015 23877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527dc6254df341948479cc8ad0f611be" candidate_uuid: "a6ba2d18b04546ec88e723b82c987e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0360707e3684c2aa75a7c82f330fce9"
I20260925 10:02:59.939016 24143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527dc6254df341948479cc8ad0f611be" candidate_uuid: "a6ba2d18b04546ec88e723b82c987e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7386dd6da6394dc4bf32ce245fa9e565"
I20260925 10:02:59.939114 24143 raft_consensus.cc:3037] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:59.939095 23877 raft_consensus.cc:3037] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:02:59.939908 23877 raft_consensus.cc:2454] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6ba2d18b04546ec88e723b82c987e54 in term 1.
I20260925 10:02:59.939960 24143 raft_consensus.cc:2454] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6ba2d18b04546ec88e723b82c987e54 in term 1.
I20260925 10:02:59.940124 23943 leader_election.cc:302] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6ba2d18b04546ec88e723b82c987e54, f0360707e3684c2aa75a7c82f330fce9; no voters:
I20260925 10:02:59.940228 24214 raft_consensus.cc:2781] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:02:59.940423 24214 raft_consensus.cc:686] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 1 LEADER]: Becoming Leader. State: Replica: a6ba2d18b04546ec88e723b82c987e54, State: Running, Role: LEADER
I20260925 10:02:59.940533 24214 consensus_queue.cc:237] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } }
I20260925 10:02:59.941576 23733 catalog_manager.cc:5495] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6ba2d18b04546ec88e723b82c987e54 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "a6ba2d18b04546ec88e723b82c987e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ba2d18b04546ec88e723b82c987e54" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:00.011637 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.008529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.19.46.194:37805)
0925 10:03:00.008598 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.008600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.008611 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:00.008661 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.008662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.008680 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.008741 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:00.008954 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.009461 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.010285 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.010486 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.010542 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:00.011197 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:00.011314 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:00.011403 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.011533 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.011555 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:03:00.013592 24143 raft_consensus.cc:1261] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 1 FOLLOWER]: Refusing update from remote peer a6ba2d18b04546ec88e723b82c987e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:00.013612 23877 raft_consensus.cc:1261] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 1 FOLLOWER]: Refusing update from remote peer a6ba2d18b04546ec88e723b82c987e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:00.013991 24214 consensus_queue.cc:1035] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7386dd6da6394dc4bf32ce245fa9e565" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:00.014364 24216 consensus_queue.cc:1035] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0360707e3684c2aa75a7c82f330fce9" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:00.017268 24221 mvcc.cc:205] Tried to move back new op lower bound from 7333194055732641792 to 7333194055437164544. Current Snapshot: MvccSnapshot[applied={T|T < 7333194055732641792}]
I20260925 10:03:00.017503 24220 mvcc.cc:205] Tried to move back new op lower bound from 7333194055732641792 to 7333194055437164544. Current Snapshot: MvccSnapshot[applied={T|T < 7333194055732641792 or (T in {7333194055732641792})}]
I20260925 10:03:00.019811 23732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:00.019958 23732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:00.020860 23732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea60eb7954e24353a4935be3e0589284: Sending DeleteTablet for 3 replicas of tablet 527dc6254df341948479cc8ad0f611be
I20260925 10:03:00.021302 23990 tablet_service.cc:1514] Processing DeleteTablet for tablet 527dc6254df341948479cc8ad0f611be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:42450
I20260925 10:03:00.021394 23857 tablet_service.cc:1514] Processing DeleteTablet for tablet 527dc6254df341948479cc8ad0f611be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:50098
I20260925 10:03:00.021690 24123 tablet_service.cc:1514] Processing DeleteTablet for tablet 527dc6254df341948479cc8ad0f611be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:59712
I20260925 10:03:00.021863 24229 tablet_replica.cc:330] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: stopping tablet replica
I20260925 10:03:00.021934 24230 tablet_replica.cc:330] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9: stopping tablet replica
I20260925 10:03:00.022051 24230 raft_consensus.cc:2229] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:00.022051 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23795
I20260925 10:03:00.022184 24230 raft_consensus.cc:2258] T 527dc6254df341948479cc8ad0f611be P f0360707e3684c2aa75a7c82f330fce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.022811 24229 raft_consensus.cc:2229] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:00.023025 24229 raft_consensus.cc:2258] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.023545 24229 ts_tablet_manager.cc:1905] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:00.025755 24229 ts_tablet_manager.cc:1918] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:00.025895 24229 log.cc:1196] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/527dc6254df341948479cc8ad0f611be
I20260925 10:03:00.026254 24229 ts_tablet_manager.cc:1939] T 527dc6254df341948479cc8ad0f611be P a6ba2d18b04546ec88e723b82c987e54: Deleting consensus metadata
I20260925 10:03:00.026687 24231 tablet_replica.cc:330] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: stopping tablet replica
I20260925 10:03:00.026749 23719 catalog_manager.cc:4841] TS a6ba2d18b04546ec88e723b82c987e54 (127.19.46.194:37805): tablet 527dc6254df341948479cc8ad0f611be (table test-table [id=ff6978ef26f5474b925e281752f51b2c]) successfully deleted
I20260925 10:03:00.026938 24231 raft_consensus.cc:2229] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:00.027086 24231 raft_consensus.cc:2258] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.027515 24231 ts_tablet_manager.cc:1905] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:00.029112 24231 ts_tablet_manager.cc:1918] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:00.029210 24231 log.cc:1196] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/527dc6254df341948479cc8ad0f611be
I20260925 10:03:00.029598 24231 ts_tablet_manager.cc:1939] T 527dc6254df341948479cc8ad0f611be P 7386dd6da6394dc4bf32ce245fa9e565: Deleting consensus metadata
I20260925 10:03:00.030055 23717 catalog_manager.cc:4841] TS 7386dd6da6394dc4bf32ce245fa9e565 (127.19.46.195:35359): tablet 527dc6254df341948479cc8ad0f611be (table test-table [id=ff6978ef26f5474b925e281752f51b2c]) successfully deleted
I20260925 10:03:00.030908 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23928
W20260925 10:03:00.030913 23717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:33395 (error 108)
I20260925 10:03:00.032792 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.032622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33395 (local address 127.0.0.1:50106)
0925 10:03:00.032696 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.032751 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:03:00.033084 23717 catalog_manager.cc:4584] TS f0360707e3684c2aa75a7c82f330fce9 (127.19.46.193:33395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 527dc6254df341948479cc8ad0f611be: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33395: connect: Connection refused (error 111)
I20260925 10:03:00.035857 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24061
I20260925 10:03:00.041038 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 23702
2026-09-25T10:03:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2282 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:00Z Disabled control of system clock
I20260925 10:03:00.069510 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38103
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:03:00.154830 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:00.155061 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:00.155084 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:00.156276 24236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:00.156322 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:00.156335 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:00.156348 24236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:00.156360 24236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:00.157871 24236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:00.158141 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:00.158460 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:00.163861 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.163892 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.163877 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.164247 24236 server_base.cc:1034] running on GCE node
I20260925 10:03:00.164554 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:00.164884 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:00.166080 24236 hybrid_clock.cc:648] HybridClock initialized: now 1790330580166060 us; error 44 us; skew 500 ppm
I20260925 10:03:00.167552 24236 webserver.cc:460] Webserver started at http://127.19.46.254:42293/ using document root <none> and password file <none>
I20260925 10:03:00.167866 24236 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:00.167943 24236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:00.168102 24236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:00.169340 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "679629cfcd4548afac475fd9af27be60"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "a0c7d2392e0ceccb2aba4c6691eec648"
server_key_iv: "8d83a4c4fe244b09f8746abc0a7db4eb"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.169996 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "679629cfcd4548afac475fd9af27be60"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "a0c7d2392e0ceccb2aba4c6691eec648"
server_key_iv: "8d83a4c4fe244b09f8746abc0a7db4eb"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.171607 24236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:03:00.172485 24250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.172719 24236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:00.172839 24236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "679629cfcd4548afac475fd9af27be60"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "a0c7d2392e0ceccb2aba4c6691eec648"
server_key_iv: "8d83a4c4fe244b09f8746abc0a7db4eb"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.172930 24236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:00.176559 24236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:00.176856 24236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:00.177008 24236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:00.181147 24236 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:38103
I20260925 10:03:00.181182 24302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:38103 every 8 connection(s)
I20260925 10:03:00.181619 24236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:00.182344 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:00.184273 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24236
I20260925 10:03:00.184435 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:00.184623 19643 external_mini_cluster.cc:1428] Setting key 8aedf8130426c6e10090664cbbc4ec62
I20260925 10:03:00.185817 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: Bootstrap starting.
I20260925 10:03:00.186563 24303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:00.186908 24303 log.cc:824] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:00.187706 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.185355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.19.46.254:38103)
0925 10:03:00.185574 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.185584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.185616 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:03:00.186195 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.186206 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.186279 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.186431 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.187226 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.187229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.187230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.187233 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.187270 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.187279 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.187345 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.187442 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.187499 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 10:03:00.187776 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: No bootstrap required, opened a new log
I20260925 10:03:00.189267 24303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } }
I20260925 10:03:00.189412 24303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:00.189491 24303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679629cfcd4548afac475fd9af27be60, State: Initialized, Role: FOLLOWER
I20260925 10:03:00.189625 24303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } }
I20260925 10:03:00.189710 24303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:00.189754 24303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:00.189808 24303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:00.190675 24303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } }
I20260925 10:03:00.190768 24303 leader_election.cc:302] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 679629cfcd4548afac475fd9af27be60; no voters:
I20260925 10:03:00.190935 24303 leader_election.cc:288] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:00.190999 24308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:00.191218 24308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [term 1 LEADER]: Becoming Leader. State: Replica: 679629cfcd4548afac475fd9af27be60, State: Running, Role: LEADER
I20260925 10:03:00.191263 24303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:00.191330 24308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } }
I20260925 10:03:00.191711 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "679629cfcd4548afac475fd9af27be60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } } }
I20260925 10:03:00.191886 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:00.192193 24310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 679629cfcd4548afac475fd9af27be60. Latest consensus state: current_term: 1 leader_uuid: "679629cfcd4548afac475fd9af27be60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679629cfcd4548afac475fd9af27be60" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38103 } } }
I20260925 10:03:00.192272 24310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:00.192490 24317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:00.192989 24317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:00.194630 24317 catalog_manager.cc:1347] Generated new cluster ID: bd0ada5631384ec3bf7608f462a48bf7
I20260925 10:03:00.194696 24317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:00.210799 24317 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:00.211054 24317 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:00.215705 24317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: Generated new TSK 0
I20260925 10:03:00.215919 24317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:00.228070 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--builtin_ntp_servers=127.19.46.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:03:00.314798 24327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:00.315032 24327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:00.315057 24327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:00.316363 24327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:00.316414 24327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:00.316444 24327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:00.318014 24327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:00.318306 24327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:00.318647 24327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:00.325153 24333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.325208 24332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.325218 24335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.325800 24327 server_base.cc:1034] running on GCE node
I20260925 10:03:00.326074 24327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:00.326359 24327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:00.327557 24327 hybrid_clock.cc:648] HybridClock initialized: now 1790330580327551 us; error 44 us; skew 500 ppm
I20260925 10:03:00.329109 24327 webserver.cc:460] Webserver started at http://127.19.46.193:34133/ using document root <none> and password file <none>
I20260925 10:03:00.329421 24327 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:00.329535 24327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:00.329706 24327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:00.330919 24327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "60ca5330c6794087b4977cd08cb99ead"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "01ef5bcf509e21485e26a51b0adb74f3"
server_key_iv: "eaf75990f175ee083aac55a8952a0665"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.331302 24327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "60ca5330c6794087b4977cd08cb99ead"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "01ef5bcf509e21485e26a51b0adb74f3"
server_key_iv: "eaf75990f175ee083aac55a8952a0665"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.332777 24327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:00.334194 24341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.334520 24327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:00.334666 24327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "60ca5330c6794087b4977cd08cb99ead"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "01ef5bcf509e21485e26a51b0adb74f3"
server_key_iv: "eaf75990f175ee083aac55a8952a0665"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.334759 24327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:00.338696 24327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:00.339015 24327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:00.339169 24327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:00.339397 24327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:00.339754 24327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:00.339811 24327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.339859 24327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:00.339879 24327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.346621 24327 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:38619
I20260925 10:03:00.346838 24454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:38619 every 8 connection(s)
I20260925 10:03:00.347136 24327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:00.351267 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.348347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38103 (local address 127.19.46.193:36899)
0925 10:03:00.348577 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.348622 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.349154 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.349426 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.349479 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.349517 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.350561 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.350561 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.350594 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.350627 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.350904 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.350912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.350915 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.351019 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.351083 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260925 10:03:00.351452 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.348366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:36899 (local address 127.19.46.254:38103)
0925 10:03:00.348465 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.348468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.348743 (+ 275us) server_negotiation.cc:407] Connection header received
0925 10:03:00.349280 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.349281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.349290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.349367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.350744 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.350745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.350746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.350750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.350775 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.350780 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.350836 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.351127 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.351311 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:03:00.351994 24455 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38103
I20260925 10:03:00.352126 24455 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:00.352289 24455 heartbeater.cc:511] Master 127.19.46.254:38103 requested a full tablet report, sending...
I20260925 10:03:00.352869 24267 ts_manager.cc:194] Registered new tserver with Master: 60ca5330c6794087b4977cd08cb99ead (127.19.46.193:38619)
I20260925 10:03:00.353896 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24327
I20260925 10:03:00.353981 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:00.354192 19643 external_mini_cluster.cc:1428] Setting key 2bc571e57ab40b62740c8f3120f15ed9
I20260925 10:03:00.355278 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--builtin_ntp_servers=127.19.46.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:03:00.447575 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:00.447873 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:00.447925 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:00.449246 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:00.449337 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:00.449385 24458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:00.450932 24458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:00.451275 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:00.451702 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:00.458228 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.458305 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.458292 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.458832 24458 server_base.cc:1034] running on GCE node
I20260925 10:03:00.459167 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:00.459435 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:00.460619 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790330580460594 us; error 41 us; skew 500 ppm
I20260925 10:03:00.462277 24458 webserver.cc:460] Webserver started at http://127.19.46.194:34047/ using document root <none> and password file <none>
I20260925 10:03:00.462594 24458 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:00.462766 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:00.462949 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:00.464165 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "e281370ebe464394b206eb0cea1b9957"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "f9c0e5694092ab8e42db9942bc813709"
server_key_iv: "7565dba0e341edce580b4b5beef1a3e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.464558 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "e281370ebe464394b206eb0cea1b9957"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "f9c0e5694092ab8e42db9942bc813709"
server_key_iv: "7565dba0e341edce580b4b5beef1a3e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.466315 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:03:00.467365 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.467793 24458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:00.468034 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "e281370ebe464394b206eb0cea1b9957"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "f9c0e5694092ab8e42db9942bc813709"
server_key_iv: "7565dba0e341edce580b4b5beef1a3e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.468269 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:00.472213 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:00.472537 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:00.472688 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:00.472935 24458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:00.473304 24458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:00.473361 24458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.473408 24458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:00.473460 24458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.480250 24458 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37675
I20260925 10:03:00.480484 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37675 every 8 connection(s)
I20260925 10:03:00.480788 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:00.485224 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.482112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:55919 (local address 127.19.46.254:38103)
0925 10:03:00.482226 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.482229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.482427 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:03:00.483025 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.483026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.483033 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.483124 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.484868 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.484870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.484870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.484876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.484894 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.484899 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.484965 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.485138 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.485156 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:03:00.485406 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.482028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38103 (local address 127.19.46.194:55919)
0925 10:03:00.482256 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.482301 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.482822 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.483333 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.483340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.483354 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.484643 (+ 1289us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.484644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.484684 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.484731 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.484980 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.484985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.484989 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.485087 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.485156 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260925 10:03:00.486274 24586 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38103
I20260925 10:03:00.486507 24586 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:00.486860 24586 heartbeater.cc:511] Master 127.19.46.254:38103 requested a full tablet report, sending...
I20260925 10:03:00.487632 24267 ts_manager.cc:194] Registered new tserver with Master: e281370ebe464394b206eb0cea1b9957 (127.19.46.194:37675)
I20260925 10:03:00.490846 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24458
I20260925 10:03:00.490954 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:00.491215 19643 external_mini_cluster.cc:1428] Setting key d3eacf436ab881a468f1b36896ab1d23
I20260925 10:03:00.492656 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--builtin_ntp_servers=127.19.46.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:03:00.580761 24589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:00.581029 24589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:00.581071 24589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:00.582784 24589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:00.582854 24589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:00.582901 24589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:00.585248 24589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:00.585580 24589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:00.585909 24589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:00.591555 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.591538 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.591657 24589 server_base.cc:1034] running on GCE node
W20260925 10:03:00.591516 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.591956 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:00.592317 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:00.593529 24589 hybrid_clock.cc:648] HybridClock initialized: now 1790330580593502 us; error 41 us; skew 500 ppm
I20260925 10:03:00.594970 24589 webserver.cc:460] Webserver started at http://127.19.46.195:37451/ using document root <none> and password file <none>
I20260925 10:03:00.595233 24589 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:00.595284 24589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:00.595450 24589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:00.596606 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "7f92477bc3324646baf7b6ead99ab9cc"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "db86523bc944da259eba5c01befb313f"
server_key_iv: "4cafbda0ae80902fa60ddd14e771c20f"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.597002 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "7f92477bc3324646baf7b6ead99ab9cc"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "db86523bc944da259eba5c01befb313f"
server_key_iv: "4cafbda0ae80902fa60ddd14e771c20f"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.598541 24589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:00.599373 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.599570 24589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:00.599668 24589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "7f92477bc3324646baf7b6ead99ab9cc"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "db86523bc944da259eba5c01befb313f"
server_key_iv: "4cafbda0ae80902fa60ddd14e771c20f"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.599746 24589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:00.603369 24589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:00.603667 24589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:00.603834 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:00.604053 24589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:00.604379 24589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:00.604413 24589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.604442 24589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:00.604490 24589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.610661 24589 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:35457
I20260925 10:03:00.610739 24716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:35457 every 8 connection(s)
I20260925 10:03:00.611107 24589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:00.614327 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.611997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:39223 (local address 127.19.46.254:38103)
0925 10:03:00.612230 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.612233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.612248 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:03:00.612739 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.612740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.612746 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.612825 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.614110 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.614112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.614112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.614116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.614132 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.614136 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.614192 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.614246 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.614258 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260925 10:03:00.614542 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.611914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38103 (local address 127.19.46.195:39223)
0925 10:03:00.612112 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.612155 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.612621 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.612877 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.612881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.612889 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.613933 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.613934 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.613964 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.613995 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.614190 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.614192 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.614195 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.614247 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.614296 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260925 10:03:00.615180 24717 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38103
I20260925 10:03:00.615334 24717 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:00.615520 24717 heartbeater.cc:511] Master 127.19.46.254:38103 requested a full tablet report, sending...
I20260925 10:03:00.615980 24267 ts_manager.cc:194] Registered new tserver with Master: 7f92477bc3324646baf7b6ead99ab9cc (127.19.46.195:35457)
I20260925 10:03:00.619771 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24589
I20260925 10:03:00.619915 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:00.620143 19643 external_mini_cluster.cc:1428] Setting key f1ac7811e36ef00fb490762b94d11b15
I20260925 10:03:00.621158 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:03:00.623823 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.622673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.19.46.254:38103)
0925 10:03:00.622769 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.622771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.622863 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:03:00.622938 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.622939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.622945 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.623004 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.623608 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.623609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.623609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.623614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.623627 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.623632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.623682 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.623728 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.623744 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:03:00.625648 24267 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:00.627261 24267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:00.631474 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:35457 (local address 127.0.0.1:57282)
0925 10:03:00.630145 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.630153 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.630202 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.630414 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.630415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.630418 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.631079 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.631079 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.631087 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.631103 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.631322 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.631322 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.631323 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.631351 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.631360 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":6,"thread_start_us":60,"threads_started":1}
I20260925 10:03:00.631613 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.19.46.193:38619)
0925 10:03:00.629878 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.629882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.630103 (+ 221us) server_negotiation.cc:407] Connection header received
0925 10:03:00.630138 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.630141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.630152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.630234 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.631039 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.631048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.631048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.631053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.631088 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.631110 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.631179 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.631326 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.631387 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260925 10:03:00.631415 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38619 (local address 127.0.0.1:55270)
0925 10:03:00.629938 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.629977 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.630041 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.630324 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.630332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.630341 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.630930 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.630930 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.630947 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.630972 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.631199 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.631205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.631208 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.631305 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.631321 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":93,"threads_started":1}
I20260925 10:03:00.631671 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.19.46.195:35457)
0925 10:03:00.629856 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.629861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.630253 (+ 392us) server_negotiation.cc:407] Connection header received
0925 10:03:00.630297 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.630300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.630312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.630412 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.631183 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.631189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.631189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.631193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.631223 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.631242 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.631478 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.631537 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.631562 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260925 10:03:00.632227 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37675 (local address 127.0.0.1:46790)
0925 10:03:00.630276 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.630284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.630319 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.630727 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.630728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.630731 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.631216 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.631216 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.631220 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.631228 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.631493 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.631493 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.631494 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.632137 (+ 643us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.632152 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":67,"threads_started":1}
I20260925 10:03:00.632511 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.629876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.19.46.194:37675)
0925 10:03:00.630048 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.630052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.630394 (+ 342us) server_negotiation.cc:407] Connection header received
0925 10:03:00.630615 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.630621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.630631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.630707 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.631310 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.631318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.631318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.631322 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.631358 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.631393 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.631453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.632404 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.632421 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260925 10:03:00.632385 24389 tablet_service.cc:1467] Processing CreateTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b (DEFAULT_TABLE table=test-table [id=fa371cefa59841888d1a4d0a5914685a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:00.632385 24651 tablet_service.cc:1467] Processing CreateTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b (DEFAULT_TABLE table=test-table [id=fa371cefa59841888d1a4d0a5914685a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:00.632763 24651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103ab744c7eb4c9cb45d607be8a2243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:00.632763 24389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103ab744c7eb4c9cb45d607be8a2243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:00.633131 24520 tablet_service.cc:1467] Processing CreateTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b (DEFAULT_TABLE table=test-table [id=fa371cefa59841888d1a4d0a5914685a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:00.633522 24520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103ab744c7eb4c9cb45d607be8a2243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:00.636111 24731 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Bootstrap starting.
I20260925 10:03:00.636471 24732 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Bootstrap starting.
I20260925 10:03:00.636914 24731 tablet_bootstrap.cc:654] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:00.637115 24732 tablet_bootstrap.cc:654] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:00.637270 24731 log.cc:824] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:00.637411 24732 log.cc:824] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:00.637753 24733 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: Bootstrap starting.
I20260925 10:03:00.638219 24731 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: No bootstrap required, opened a new log
I20260925 10:03:00.638295 24731 ts_tablet_manager.cc:1397] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:03:00.638410 24733 tablet_bootstrap.cc:654] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:00.638682 24733 log.cc:824] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:00.639230 24732 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: No bootstrap required, opened a new log
I20260925 10:03:00.639315 24732 ts_tablet_manager.cc:1397] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:03:00.639451 24733 tablet_bootstrap.cc:492] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: No bootstrap required, opened a new log
I20260925 10:03:00.639523 24733 ts_tablet_manager.cc:1397] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:03:00.640952 24731 raft_consensus.cc:348] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.641140 24731 raft_consensus.cc:374] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:00.641176 24731 raft_consensus.cc:729] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e281370ebe464394b206eb0cea1b9957, State: Initialized, Role: FOLLOWER
I20260925 10:03:00.641227 24732 raft_consensus.cc:348] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.641330 24731 consensus_queue.cc:260] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.641387 24732 raft_consensus.cc:374] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:00.641456 24732 raft_consensus.cc:729] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f92477bc3324646baf7b6ead99ab9cc, State: Initialized, Role: FOLLOWER
I20260925 10:03:00.641597 24732 consensus_queue.cc:260] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.641672 24731 ts_tablet_manager.cc:1428] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:03:00.641860 24732 ts_tablet_manager.cc:1428] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:00.641917 24586 heartbeater.cc:503] Master 127.19.46.254:38103 was elected leader, sending a full tablet report...
I20260925 10:03:00.641971 24717 heartbeater.cc:503] Master 127.19.46.254:38103 was elected leader, sending a full tablet report...
I20260925 10:03:00.642323 24733 raft_consensus.cc:348] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.642493 24733 raft_consensus.cc:374] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:00.642529 24733 raft_consensus.cc:729] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ca5330c6794087b4977cd08cb99ead, State: Initialized, Role: FOLLOWER
I20260925 10:03:00.642640 24733 consensus_queue.cc:260] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.642921 24733 ts_tablet_manager.cc:1428] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:03:00.643088 24455 heartbeater.cc:503] Master 127.19.46.254:38103 was elected leader, sending a full tablet report...
I20260925 10:03:00.644832 24738 raft_consensus.cc:482] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:00.644920 24738 raft_consensus.cc:504] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.645219 24738 leader_election.cc:288] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ca5330c6794087b4977cd08cb99ead (127.19.46.193:38619), e281370ebe464394b206eb0cea1b9957 (127.19.46.194:37675)
I20260925 10:03:00.646576 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.645402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38619 (local address 127.19.46.195:46559)
0925 10:03:00.645485 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.645499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.645544 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.645707 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.645709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.645712 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.646133 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.646133 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.646139 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.646150 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.646430 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.646430 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.646431 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.646488 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.646502 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 10:03:00.646592 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.645473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:46559 (local address 127.19.46.193:38619)
0925 10:03:00.645523 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.645525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.645533 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:00.645621 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.645621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.645626 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.645675 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.646348 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.646349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.646350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.646354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.646371 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.646375 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.646434 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.646487 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.646502 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:03:00.646632 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.645491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:59689 (local address 127.19.46.194:37675)
0925 10:03:00.645569 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.645571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.645724 (+ 153us) server_negotiation.cc:407] Connection header received
0925 10:03:00.645751 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.645752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.645756 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.645815 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.646378 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.646379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.646380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.646384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.646398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.646402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.646455 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.646547 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.646560 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:03:00.646821 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.645402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37675 (local address 127.19.46.195:59689)
0925 10:03:00.645626 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.645636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.645708 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.645822 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.645822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.645825 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:00.646290 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:00.646290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:00.646295 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:00.646309 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:00.646479 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:00.646479 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:00.646480 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:03:00.646730 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 10:03:00.646742 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":80,"thread_start_us":80,"threads_started":1}
I20260925 10:03:00.646893 24409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103ab744c7eb4c9cb45d607be8a2243b" candidate_uuid: "7f92477bc3324646baf7b6ead99ab9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ca5330c6794087b4977cd08cb99ead" is_pre_election: true
I20260925 10:03:00.647038 24409 raft_consensus.cc:2454] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f92477bc3324646baf7b6ead99ab9cc in term 0.
I20260925 10:03:00.647171 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103ab744c7eb4c9cb45d607be8a2243b" candidate_uuid: "7f92477bc3324646baf7b6ead99ab9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e281370ebe464394b206eb0cea1b9957" is_pre_election: true
I20260925 10:03:00.647234 24604 leader_election.cc:302] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ca5330c6794087b4977cd08cb99ead, 7f92477bc3324646baf7b6ead99ab9cc; no voters:
I20260925 10:03:00.647307 24540 raft_consensus.cc:2454] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f92477bc3324646baf7b6ead99ab9cc in term 0.
I20260925 10:03:00.647339 24738 raft_consensus.cc:2781] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:00.647392 24738 raft_consensus.cc:482] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:00.647420 24738 raft_consensus.cc:3037] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:00.648193 24738 raft_consensus.cc:504] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.648334 24738 leader_election.cc:288] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [CANDIDATE]: Term 1 election: Requested vote from peers 60ca5330c6794087b4977cd08cb99ead (127.19.46.193:38619), e281370ebe464394b206eb0cea1b9957 (127.19.46.194:37675)
I20260925 10:03:00.648461 24409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103ab744c7eb4c9cb45d607be8a2243b" candidate_uuid: "7f92477bc3324646baf7b6ead99ab9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ca5330c6794087b4977cd08cb99ead"
I20260925 10:03:00.648554 24409 raft_consensus.cc:3037] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:00.648602 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103ab744c7eb4c9cb45d607be8a2243b" candidate_uuid: "7f92477bc3324646baf7b6ead99ab9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e281370ebe464394b206eb0cea1b9957"
I20260925 10:03:00.648679 24540 raft_consensus.cc:3037] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:00.649257 24409 raft_consensus.cc:2454] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f92477bc3324646baf7b6ead99ab9cc in term 1.
I20260925 10:03:00.649418 24540 raft_consensus.cc:2454] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f92477bc3324646baf7b6ead99ab9cc in term 1.
I20260925 10:03:00.649498 24604 leader_election.cc:302] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ca5330c6794087b4977cd08cb99ead, 7f92477bc3324646baf7b6ead99ab9cc; no voters:
I20260925 10:03:00.649602 24738 raft_consensus.cc:2781] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:00.649736 24738 raft_consensus.cc:686] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 1 LEADER]: Becoming Leader. State: Replica: 7f92477bc3324646baf7b6ead99ab9cc, State: Running, Role: LEADER
I20260925 10:03:00.649840 24738 consensus_queue.cc:237] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } }
I20260925 10:03:00.650779 24267 catalog_manager.cc:5495] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f92477bc3324646baf7b6ead99ab9cc (127.19.46.195). New cstate: current_term: 1 leader_uuid: "7f92477bc3324646baf7b6ead99ab9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f92477bc3324646baf7b6ead99ab9cc" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 35457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:00.656029 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.654752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.19.46.195:35457)
0925 10:03:00.654872 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.654873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.654882 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:00.654923 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.654924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.654939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.654996 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.655684 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.655685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.655686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.655689 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.655704 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.655708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.655772 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.655881 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.655893 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:03:00.657992 24409 raft_consensus.cc:1261] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 1 FOLLOWER]: Refusing update from remote peer 7f92477bc3324646baf7b6ead99ab9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:00.658074 24540 raft_consensus.cc:1261] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 1 FOLLOWER]: Refusing update from remote peer 7f92477bc3324646baf7b6ead99ab9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:00.658437 24741 consensus_queue.cc:1035] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e281370ebe464394b206eb0cea1b9957" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:00.658567 24738 consensus_queue.cc:1035] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ca5330c6794087b4977cd08cb99ead" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:00.663388 24746 mvcc.cc:205] Tried to move back new op lower bound from 7333194058371858432 to 7333194058342600704. Current Snapshot: MvccSnapshot[applied={T|T < 7333194058371858432 or (T in {7333194058371858432})}]
I20260925 10:03:00.663401 24266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:00.663502 24266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:00.664397 24266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 679629cfcd4548afac475fd9af27be60: Sending DeleteTablet for 3 replicas of tablet 103ab744c7eb4c9cb45d607be8a2243b
I20260925 10:03:00.664754 24651 tablet_service.cc:1514] Processing DeleteTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='slave'} at 127.0.0.1:57282
I20260925 10:03:00.664835 24520 tablet_service.cc:1514] Processing DeleteTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='slave'} at 127.0.0.1:46790
I20260925 10:03:00.665015 24389 tablet_service.cc:1514] Processing DeleteTablet for tablet 103ab744c7eb4c9cb45d607be8a2243b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:00 UTC) from {username='slave'} at 127.0.0.1:55270
I20260925 10:03:00.665128 24754 tablet_replica.cc:330] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: stopping tablet replica
I20260925 10:03:00.665247 24754 raft_consensus.cc:2229] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:00.665342 24755 tablet_replica.cc:330] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead: stopping tablet replica
I20260925 10:03:00.665378 24754 raft_consensus.cc:2258] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.665654 24755 raft_consensus.cc:2229] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:00.665727 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24327
I20260925 10:03:00.665861 24755 raft_consensus.cc:2258] T 103ab744c7eb4c9cb45d607be8a2243b P 60ca5330c6794087b4977cd08cb99ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.666803 24754 ts_tablet_manager.cc:1905] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:00.669042 24754 ts_tablet_manager.cc:1918] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:00.669133 24754 log.cc:1196] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/103ab744c7eb4c9cb45d607be8a2243b
I20260925 10:03:00.669456 24754 ts_tablet_manager.cc:1939] T 103ab744c7eb4c9cb45d607be8a2243b P e281370ebe464394b206eb0cea1b9957: Deleting consensus metadata
I20260925 10:03:00.669998 24253 catalog_manager.cc:4841] TS e281370ebe464394b206eb0cea1b9957 (127.19.46.194:37675): tablet 103ab744c7eb4c9cb45d607be8a2243b (table test-table [id=fa371cefa59841888d1a4d0a5914685a]) successfully deleted
W20260925 10:03:00.674424 24251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:38619 (error 108)
I20260925 10:03:00.674491 24753 tablet_replica.cc:330] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: stopping tablet replica
I20260925 10:03:00.674656 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24458
I20260925 10:03:00.674857 24753 raft_consensus.cc:2229] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:00.675050 24753 raft_consensus.cc:2258] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:00.675694 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.675514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38619 (local address 127.0.0.1:55278)
0925 10:03:00.675591 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.675647 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:03:00.675884 24251 catalog_manager.cc:4584] TS 60ca5330c6794087b4977cd08cb99ead (127.19.46.193:38619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 103ab744c7eb4c9cb45d607be8a2243b: Network error: Client connection negotiation failed: client connection to 127.19.46.193:38619: connect: Connection refused (error 111)
I20260925 10:03:00.680200 24753 ts_tablet_manager.cc:1905] T 103ab744c7eb4c9cb45d607be8a2243b P 7f92477bc3324646baf7b6ead99ab9cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:00.681053 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24589
I20260925 10:03:00.686479 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24236
I20260925 10:03:00.686597 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.686453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:35457 (local address 127.0.0.1:57304)
0925 10:03:00.686521 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.686549 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.195:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
2026-09-25T10:03:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (645 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:03:00.708675 19643 mini_oidc.cc:152] Starting JWKS server
I20260925 10:03:00.710371 19643 webserver.cc:460] Webserver started at https://127.0.0.1:43557/ using document root <none> and password file <none>
I20260925 10:03:00.710469 19643 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:03:00.710676 19643 webserver.cc:460] Webserver started at http://127.0.0.1:46805/ using document root <none> and password file <none>
2026-09-25T10:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:00Z Disabled control of system clock
I20260925 10:03:00.720309 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46465
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/testchainca.cert with env {}
W20260925 10:03:00.808199 24762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:00.808439 24762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:00.808465 24762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:00.809767 24762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:00.809836 24762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:00.809849 24762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:00.809862 24762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:00.809875 24762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:03:00.809887 24762 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:03:00.809898 24762 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43557/jwks.json
I20260925 10:03:00.812354 24762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:00.812624 24762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:00.812816 24762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:03:00.818837 24762 server_base.cc:1034] running on GCE node
W20260925 10:03:00.818814 24770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.818814 24767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:00.818761 24768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:00.819453 24762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:00.819952 24762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:00.821180 24762 hybrid_clock.cc:648] HybridClock initialized: now 1790330580821137 us; error 65 us; skew 500 ppm
I20260925 10:03:00.822645 24762 webserver.cc:460] Webserver started at http://127.19.46.254:35315/ using document root <none> and password file <none>
I20260925 10:03:00.822919 24762 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:00.822971 24762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:00.823153 24762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:00.824342 24762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "d73647ca074e4d71b366c4de4fbb666a"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "59d5a3fe1477e258d6bceb285e7731e4"
server_key_iv: "402d058d1c2d7e6c257ba35b8a21c5df"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.824735 24762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "d73647ca074e4d71b366c4de4fbb666a"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "59d5a3fe1477e258d6bceb285e7731e4"
server_key_iv: "402d058d1c2d7e6c257ba35b8a21c5df"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.826342 24762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:00.827329 24776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:00.827622 24762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:00.827771 24762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "d73647ca074e4d71b366c4de4fbb666a"
format_stamp: "Formatted at 2026-09-25 10:03:00 on dist-test-slave-f9rl"
server_key: "59d5a3fe1477e258d6bceb285e7731e4"
server_key_iv: "402d058d1c2d7e6c257ba35b8a21c5df"
server_key_version: "encryptionkey@0"
I20260925 10:03:00.827862 24762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:00.839469 24762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:00.839864 24762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:00.840034 24762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:00.844659 24762 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46465
I20260925 10:03:00.844696 24828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46465 every 8 connection(s)
I20260925 10:03:00.845096 24762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:00.845959 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24762
I20260925 10:03:00.846143 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:00.846249 24829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:00.846422 19643 external_mini_cluster.cc:1428] Setting key 73ff89d43e5dc872fc96c102745d1bce
I20260925 10:03:00.850004 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a: Bootstrap starting.
I20260925 10:03:00.850701 24829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:00.851008 24829 log.cc:824] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:00.851753 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a: No bootstrap required, opened a new log
I20260925 10:03:00.853152 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.847217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.19.46.254:46465)
0925 10:03:00.847426 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.847435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.847471 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:03:00.847982 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.847992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.848091 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.848256 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:00.848502 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.849900 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.850844 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.851330 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.851496 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:00.852472 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:00.852477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:00.852478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:00.852668 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:00.852703 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:00.852715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:00.852795 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:00.852880 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:03:00.852962 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260925 10:03:00.853899 24829 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } }
I20260925 10:03:00.854091 24829 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:00.854157 24829 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73647ca074e4d71b366c4de4fbb666a, State: Initialized, Role: FOLLOWER
I20260925 10:03:00.854269 24829 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } }
I20260925 10:03:00.854333 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:03:00.854355 24829 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:00.854400 24829 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:00.854451 24829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:00.855041 24829 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } }
I20260925 10:03:00.855168 24829 leader_election.cc:302] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73647ca074e4d71b366c4de4fbb666a; no voters:
I20260925 10:03:00.855365 24829 leader_election.cc:288] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:00.855399 24834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:00.855540 24834 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [term 1 LEADER]: Becoming Leader. State: Replica: d73647ca074e4d71b366c4de4fbb666a, State: Running, Role: LEADER
I20260925 10:03:00.855638 24834 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } }
I20260925 10:03:00.855690 24829 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:00.856040 24835 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73647ca074e4d71b366c4de4fbb666a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } } }
I20260925 10:03:00.856076 24836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73647ca074e4d71b366c4de4fbb666a. Latest consensus state: current_term: 1 leader_uuid: "d73647ca074e4d71b366c4de4fbb666a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73647ca074e4d71b366c4de4fbb666a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46465 } } }
I20260925 10:03:00.856282 24835 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:00.856300 24836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:00.856633 24843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:00.857161 24843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:00.859566 24843 catalog_manager.cc:1347] Generated new cluster ID: c07a3a5859f14f498b4b57c6fd83c7a7
I20260925 10:03:00.859648 24843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:00.880551 24843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:00.881138 24843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:00.889642 24843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73647ca074e4d71b366c4de4fbb666a: Generated new TSK 0
I20260925 10:03:00.889897 24843 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:03:00.915625 24859 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:03:00.915747 24859 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 10:03:00.916260 24858 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:00.899583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46465 (local address 127.0.0.1:52620)
0925 10:03:00.899762 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:00.899774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:00.899879 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:00.900267 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:00.900270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:00.900293 (+ 23us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:03:00.900407 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:00.900411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.901386 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.901387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:00.902248 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:00.902257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.902803 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.902804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:00.902876 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:00.902913 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:03:00.915908 (+ 12995us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:03:00.916007 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.46.254:46465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.46.254:46465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:03:00.916601 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:00.899664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.19.46.254:46465)
0925 10:03:00.899719 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:00.899722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:00.899853 (+ 131us) server_negotiation.cc:407] Connection header received
0925 10:03:00.900024 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:00.900025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:00.900039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:00.900107 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:03:00.900558 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.901235 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.902448 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:00.902706 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:00.902799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:00.903056 (+ 257us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:03:00.915741 (+ 12685us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:03:00.916535 (+ 794us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52620: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 10:03:00.916646 24832 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52620: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260925 10:03:00.916654 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24762
2026-09-25T10:03:00Z chronyd exiting
I20260925 10:03:00.953773 19643 test_util.cc:175] -----------------------------------------------
I20260925 10:03:00.953866 19643 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330556283750-19643-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:03:00.954756 19643 mini_oidc.cc:152] Starting JWKS server
I20260925 10:03:00.955492 19643 webserver.cc:460] Webserver started at https://127.0.0.1:44489/ using document root <none> and password file <none>
I20260925 10:03:00.955538 19643 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:03:00.955703 19643 webserver.cc:460] Webserver started at http://127.0.0.1:38217/ using document root <none> and password file <none>
2026-09-25T10:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:00Z Disabled control of system clock
I20260925 10:03:00.964170 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42269
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:42269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/testchainca.cert with env {}
W20260925 10:03:01.048731 24865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:01.049031 24865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:01.049074 24865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:01.050359 24865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:01.050443 24865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:01.050477 24865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:01.050508 24865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:01.050534 24865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:03:01.050562 24865 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:03:01.050585 24865 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44489/jwks.json
I20260925 10:03:01.052666 24865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:42269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:01.052989 24865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:01.053196 24865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:01.058427 24871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:01.058490 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:01.058462 24870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:01.058665 24865 server_base.cc:1034] running on GCE node
I20260925 10:03:01.058923 24865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:01.059237 24865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:01.060431 24865 hybrid_clock.cc:648] HybridClock initialized: now 1790330581060407 us; error 39 us; skew 500 ppm
I20260925 10:03:01.061833 24865 webserver.cc:460] Webserver started at http://127.19.46.254:39291/ using document root <none> and password file <none>
I20260925 10:03:01.062105 24865 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:01.062177 24865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:01.062325 24865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:01.063488 24865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "8295898952ea4eb0aa55b075ffc23d7e"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "611fbdb3ca0a9d01b9d06ea20e1063db"
server_key_iv: "00dff60d8aa1147c3d8a90ade1e73789"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.063867 24865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "8295898952ea4eb0aa55b075ffc23d7e"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "611fbdb3ca0a9d01b9d06ea20e1063db"
server_key_iv: "00dff60d8aa1147c3d8a90ade1e73789"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.065271 24865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:01.066527 24879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:01.066706 24865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:01.066810 24865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "8295898952ea4eb0aa55b075ffc23d7e"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "611fbdb3ca0a9d01b9d06ea20e1063db"
server_key_iv: "00dff60d8aa1147c3d8a90ade1e73789"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.066900 24865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:01.089936 24865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:01.090327 24865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:01.090499 24865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:01.095181 24865 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:42269
I20260925 10:03:01.095254 24931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:42269 every 8 connection(s)
I20260925 10:03:01.095598 24865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:01.096436 24932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:01.099478 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24865
I20260925 10:03:01.099602 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:01.099777 19643 external_mini_cluster.cc:1428] Setting key 4b359799e020b72b93fa4488243a49f1
I20260925 10:03:01.100271 24932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e: Bootstrap starting.
I20260925 10:03:01.100962 24932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:01.101274 24932 log.cc:824] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:01.102108 24932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e: No bootstrap required, opened a new log
I20260925 10:03:01.104388 24932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } }
I20260925 10:03:01.104589 24932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:01.104630 24932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8295898952ea4eb0aa55b075ffc23d7e, State: Initialized, Role: FOLLOWER
I20260925 10:03:01.104803 24932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } }
I20260925 10:03:01.104907 24932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:01.104934 24932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:01.105005 24932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:01.105897 24932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } }
I20260925 10:03:01.106017 24932 leader_election.cc:302] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8295898952ea4eb0aa55b075ffc23d7e; no voters:
I20260925 10:03:01.106189 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.100336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.19.46.254:42269)
0925 10:03:01.100530 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.100539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.100573 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:03:01.101100 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.101109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.101200 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.101329 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:01.101638 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.102680 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.103888 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.104461 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.104572 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.105383 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:01.105387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:01.105387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:01.105637 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:01.105671 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:01.105683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:01.105737 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:01.105845 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:03:01.105915 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260925 10:03:01.106211 24932 leader_election.cc:288] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:01.106274 24937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:01.106412 24937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [term 1 LEADER]: Becoming Leader. State: Replica: 8295898952ea4eb0aa55b075ffc23d7e, State: Running, Role: LEADER
I20260925 10:03:01.106498 24937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } }
I20260925 10:03:01.106542 24932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:01.106848 24937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8295898952ea4eb0aa55b075ffc23d7e. Latest consensus state: current_term: 1 leader_uuid: "8295898952ea4eb0aa55b075ffc23d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } } }
I20260925 10:03:01.106854 24938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8295898952ea4eb0aa55b075ffc23d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8295898952ea4eb0aa55b075ffc23d7e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42269 } } }
I20260925 10:03:01.106930 24937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:01.106957 24938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:01.107371 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:03:01.107728 24942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:01.108247 24942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:01.110020 24942 catalog_manager.cc:1347] Generated new cluster ID: d4345ea6b316429293fc4e3f5c0c8e8f
I20260925 10:03:01.110100 24942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:01.117550 24942 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:01.118330 24942 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:01.123004 24942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8295898952ea4eb0aa55b075ffc23d7e: Generated new TSK 0
I20260925 10:03:01.123286 24942 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:03:01.146312 24962 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:03:01.146432 24962 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:03:01.146797 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.129550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.19.46.254:42269)
0925 10:03:01.129601 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.129603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.129725 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:03:01.129788 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.129789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.129800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.129866 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:03:01.130066 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.130712 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.131973 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.132313 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.132425 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.132699 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:03:01.146581 (+ 13882us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:03:01.146736 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 10:03:01.146880 24935 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 10:03:01.147025 24961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:01.129418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42269 (local address 127.0.0.1:48592)
0925 10:03:01.129620 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:01.129633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:01.129715 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:01.129889 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:01.129890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:01.129893 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:03:01.129962 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.129964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.130826 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.130827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.131812 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.131815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.132446 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.132447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.132528 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.132534 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:03:01.146735 (+ 14201us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:03:01.146909 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.46.254:42269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":137,"threads_started":1}
I20260925 10:03:01.147486 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24865
2026-09-25T10:03:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:03:01.186558 19643 mini_oidc.cc:152] Starting JWKS server
I20260925 10:03:01.187256 19643 webserver.cc:460] Webserver started at https://127.0.0.1:43727/ using document root <none> and password file <none>
I20260925 10:03:01.187297 19643 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:03:01.187450 19643 webserver.cc:460] Webserver started at http://127.0.0.1:46115/ using document root <none> and password file <none>
2026-09-25T10:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:01Z Disabled control of system clock
I20260925 10:03:01.195379 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39257
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:39257
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43727/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:03:01.276221 24968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:01.276448 24968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:01.276470 24968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:01.277727 24968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:01.277770 24968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:01.277783 24968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:01.277796 24968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:01.277808 24968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:03:01.277849 24968 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:03:01.277874 24968 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43727/jwks.json
I20260925 10:03:01.279275 24968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:39257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43727/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:01.279556 24968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:01.279860 24968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:01.285220 24974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:01.285233 24973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:01.285220 24976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:01.285663 24968 server_base.cc:1034] running on GCE node
I20260925 10:03:01.285948 24968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:01.286270 24968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:01.287459 24968 hybrid_clock.cc:648] HybridClock initialized: now 1790330581287430 us; error 47 us; skew 500 ppm
I20260925 10:03:01.288806 24968 webserver.cc:460] Webserver started at http://127.19.46.254:35543/ using document root <none> and password file <none>
I20260925 10:03:01.289084 24968 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:01.289160 24968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:01.289310 24968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:01.290499 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "1972d09d17304881bc5a22903c290a42"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "63eecab054670e79b4b1988af7f9438b"
server_key_iv: "00ceb5bdcf847a5602690e617be21c50"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.290877 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "1972d09d17304881bc5a22903c290a42"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "63eecab054670e79b4b1988af7f9438b"
server_key_iv: "00ceb5bdcf847a5602690e617be21c50"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.292218 24968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:03:01.293071 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:01.293269 24968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:01.293372 24968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "1972d09d17304881bc5a22903c290a42"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "63eecab054670e79b4b1988af7f9438b"
server_key_iv: "00ceb5bdcf847a5602690e617be21c50"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.293503 24968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:01.300362 24968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:01.300673 24968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:01.300822 24968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:01.304816 24968 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:39257
I20260925 10:03:01.304849 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:39257 every 8 connection(s)
I20260925 10:03:01.305191 24968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:01.305979 25035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:01.308773 25035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42: Bootstrap starting.
I20260925 10:03:01.309372 25035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:01.309717 25035 log.cc:824] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:01.309945 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 24968
I20260925 10:03:01.310039 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:01.310257 19643 external_mini_cluster.cc:1428] Setting key 49c4e09a7e4d24539e9bb2a0ddd369a1
I20260925 10:03:01.310611 25035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42: No bootstrap required, opened a new log
I20260925 10:03:01.312338 25035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } }
I20260925 10:03:01.312510 25035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:01.312554 25035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1972d09d17304881bc5a22903c290a42, State: Initialized, Role: FOLLOWER
I20260925 10:03:01.312682 25035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } }
I20260925 10:03:01.312781 25035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:01.312825 25035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:01.312884 25035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:01.313705 25035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } }
I20260925 10:03:01.313817 25035 leader_election.cc:302] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1972d09d17304881bc5a22903c290a42; no voters:
I20260925 10:03:01.313977 25035 leader_election.cc:288] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:01.314073 25040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:01.314224 25040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [term 1 LEADER]: Becoming Leader. State: Replica: 1972d09d17304881bc5a22903c290a42, State: Running, Role: LEADER
I20260925 10:03:01.314226 25035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:01.314333 25040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } }
I20260925 10:03:01.314672 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1972d09d17304881bc5a22903c290a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } } }
I20260925 10:03:01.314664 25040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1972d09d17304881bc5a22903c290a42. Latest consensus state: current_term: 1 leader_uuid: "1972d09d17304881bc5a22903c290a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972d09d17304881bc5a22903c290a42" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39257 } } }
I20260925 10:03:01.314764 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:01.314774 25040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:01.315014 25047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:01.315523 25047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:01.317392 25047 catalog_manager.cc:1347] Generated new cluster ID: 438d675a9eed41708da76b47d7472190
I20260925 10:03:01.317467 25047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:01.318138 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.310759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.19.46.254:39257)
0925 10:03:01.311017 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.311024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.311048 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:03:01.312339 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.312347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.312434 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.312620 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:01.312915 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.314772 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.315980 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.316493 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.316640 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.317362 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:01.317366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:01.317366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:01.317608 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:01.317653 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:01.317667 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:01.317731 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:01.317841 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:03:01.317918 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260925 10:03:01.319124 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:03:01.330782 25047 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:01.331380 25047 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:01.335856 25047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1972d09d17304881bc5a22903c290a42: Generated new TSK 0
I20260925 10:03:01.336117 25047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:01.347051 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.342777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.19.46.254:39257)
0925 10:03:01.342849 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.342851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.342892 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:03:01.343002 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.343003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.343017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.343099 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:03:01.343278 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.344120 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.345101 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.345412 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.345521 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.345719 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:03:01.346813 (+ 1094us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43727/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:03:01.346967 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43727/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 10:03:01.347266 25064 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:01.342632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39257 (local address 127.0.0.1:36684)
0925 10:03:01.342824 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:01.342837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:01.342909 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:01.343129 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:01.343131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:01.343133 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:03:01.343198 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.343199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.344224 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.344225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.345003 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.345006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.345534 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.345535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.345586 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.345591 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:03:01.346957 (+ 1366us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43727/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:03:01.347127 (+ 170us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.46.254:39257: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43727/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
W20260925 10:03:01.347534 25039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43727/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 10:03:01.349334 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 24968
E20260925 10:03:01.359993 25065 webserver.cc:533] Webserver: sslize failed:
E20260925 10:03:01.360105 25065 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T10:03:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (201 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:01Z Disabled control of system clock
I20260925 10:03:01.396343 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44207
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:44207
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/testchainca.cert with env {}
W20260925 10:03:01.478833 25069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:01.479059 25069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:01.479082 25069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:01.480229 25069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:01.480269 25069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:01.480283 25069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:01.480298 25069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:01.480356 25069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:03:01.480396 25069 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:03:01.480417 25069 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:03:01.482173 25069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:44207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:01.482403 25069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:01.482570 25069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:01.487534 25075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:01.487608 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:01.487710 25069 server_base.cc:1034] running on GCE node
W20260925 10:03:01.487530 25077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:01.488068 25069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:01.488384 25069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:01.489547 25069 hybrid_clock.cc:648] HybridClock initialized: now 1790330581489496 us; error 55 us; skew 500 ppm
I20260925 10:03:01.490886 25069 webserver.cc:460] Webserver started at http://127.19.46.254:35869/ using document root <none> and password file <none>
I20260925 10:03:01.491154 25069 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:01.491206 25069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:01.491379 25069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:01.492534 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "9615811c7c4d4cb1bebb3debae314d55"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "7d8b7981e424590f5cc5d5df5eb0d6a1"
server_key_iv: "6817c448a96ff7a617854e93b068a965"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.492918 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "9615811c7c4d4cb1bebb3debae314d55"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "7d8b7981e424590f5cc5d5df5eb0d6a1"
server_key_iv: "6817c448a96ff7a617854e93b068a965"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.494398 25069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:01.495256 25083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:01.495465 25069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:01.495581 25069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "9615811c7c4d4cb1bebb3debae314d55"
format_stamp: "Formatted at 2026-09-25 10:03:01 on dist-test-slave-f9rl"
server_key: "7d8b7981e424590f5cc5d5df5eb0d6a1"
server_key_iv: "6817c448a96ff7a617854e93b068a965"
server_key_version: "encryptionkey@0"
I20260925 10:03:01.495664 25069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:01.533607 25069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:01.533949 25069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:01.534116 25069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:01.538002 25069 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:44207
I20260925 10:03:01.538084 25135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:44207 every 8 connection(s)
I20260925 10:03:01.538502 25069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:01.539508 25136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:01.541594 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25069
I20260925 10:03:01.541698 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:01.541910 19643 external_mini_cluster.cc:1428] Setting key 57a153abce0e732576effff5749afc8b
I20260925 10:03:01.542019 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: Bootstrap starting.
I20260925 10:03:01.542769 25136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:01.543074 25136 log.cc:824] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:01.543879 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: No bootstrap required, opened a new log
I20260925 10:03:01.546085 25136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } }
I20260925 10:03:01.546222 25136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:01.546244 25136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9615811c7c4d4cb1bebb3debae314d55, State: Initialized, Role: FOLLOWER
I20260925 10:03:01.546383 25136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } }
I20260925 10:03:01.546464 25136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:01.546509 25136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:01.546558 25136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:01.547281 25136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } }
I20260925 10:03:01.547371 25136 leader_election.cc:302] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9615811c7c4d4cb1bebb3debae314d55; no voters:
I20260925 10:03:01.547608 25136 leader_election.cc:288] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:01.547727 25141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:01.547900 25141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [term 1 LEADER]: Becoming Leader. State: Replica: 9615811c7c4d4cb1bebb3debae314d55, State: Running, Role: LEADER
I20260925 10:03:01.547895 25136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:01.548049 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.542452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.19.46.254:44207)
0925 10:03:01.542692 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.542700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.542725 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:03:01.543340 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.543349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.543427 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.543581 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:01.543794 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.544723 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.545914 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.546424 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.546574 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.547218 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:01.547222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:01.547222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:01.547435 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:01.547479 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:01.547494 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:01.547575 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:01.547685 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:03:01.547774 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260925 10:03:01.549135 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:03:01.549415 25141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } }
I20260925 10:03:01.550050 25143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9615811c7c4d4cb1bebb3debae314d55. Latest consensus state: current_term: 1 leader_uuid: "9615811c7c4d4cb1bebb3debae314d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } } }
I20260925 10:03:01.550371 25143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:01.550182 25142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9615811c7c4d4cb1bebb3debae314d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9615811c7c4d4cb1bebb3debae314d55" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44207 } } }
I20260925 10:03:01.550526 25142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55 [sys.catalog]: This master's current role is: LEADER
W20260925 10:03:01.550534 25155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:03:01.550611 25155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:03:01.550670 25158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:01.550836 25158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:01.552453 25158 catalog_manager.cc:1347] Generated new cluster ID: b71f6818c7ad4bb69e51228ee5cea6f4
I20260925 10:03:01.552520 25158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:01.572710 25158 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:01.574183 25158 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:01.580801 25158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9615811c7c4d4cb1bebb3debae314d55: Generated new TSK 0
I20260925 10:03:01.581140 25158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:01.649796 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:01.588892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.19.46.254:44207)
0925 10:03:01.589259 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:01.589261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:01.589275 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:03:01.589342 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:01.589343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:01.589358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:01.589490 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:03:01.589691 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.590521 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.591767 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.591983 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.592078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.592280 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:03:01.649571 (+ 57291us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:03:01.649734 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38984: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":338}
W20260925 10:03:01.649888 25139 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38984: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 10:03:01.650000 25165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:01.588872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44207 (local address 127.0.0.1:38984)
0925 10:03:01.589126 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:01.589138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:01.589233 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:01.589517 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:01.589519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:01.589521 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:03:01.589596 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.589598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.590610 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.590611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.591632 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:01.591635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:01.592097 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:01.592098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:01.592167 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:01.592173 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:03:01.649722 (+ 57549us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:03:01.649888 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.46.254:44207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":147,"threads_started":1}
I20260925 10:03:01.650326 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25069
2026-09-25T10:03:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (280 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (94 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:01 dist-test-slave-f9rl krb5kdc[25177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:01 dist-test-slave-f9rl krb5kdc[25177](info): set up 2 sockets
Sep 25 10:03:01 dist-test-slave-f9rl krb5kdc[25177](info): commencing operation
krb5kdc: starting...
W20260925 10:03:02.884053 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:02 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:03:03.009084 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36925
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:36925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:03.095083 25190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:03.095302 25190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:03.095328 25190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:03.096546 25190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:03.096587 25190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:03.096602 25190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:03.096614 25190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:03.096626 25190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:03.098204 25190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:36925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:03.098431 25190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:03.098635 25190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:03.103716 25195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.103958 25190 server_base.cc:1034] running on GCE node
W20260925 10:03:03.103708 25198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.103797 25196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.104311 25190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:03.104626 25190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:03.105800 25190 hybrid_clock.cc:648] HybridClock initialized: now 1790330583105784 us; error 43 us; skew 500 ppm
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.116021 25190 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:03:03.116568 25190 webserver.cc:460] Webserver started at http://127.19.46.254:40021/ using document root <none> and password file <none>
I20260925 10:03:03.116820 25190 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:03.116891 25190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:03.117051 25190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:03.118253 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "ea834d13b1db4e0e81f181046154945a"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "7ea627c75acd30842e5862b2d9f4844f"
server_key_iv: "a292f2240f9822656c5e6b76cb94854a"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.118638 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "ea834d13b1db4e0e81f181046154945a"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "7ea627c75acd30842e5862b2d9f4844f"
server_key_iv: "a292f2240f9822656c5e6b76cb94854a"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.120460 25190 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:03:03.121505 25205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.121701 25190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:03.121815 25190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "ea834d13b1db4e0e81f181046154945a"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "7ea627c75acd30842e5862b2d9f4844f"
server_key_iv: "a292f2240f9822656c5e6b76cb94854a"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.121917 25190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:03.139676 25190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:03.141981 25190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:03.142155 25190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:03.146306 25190 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:36925
I20260925 10:03:03.146354 25257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:36925 every 8 connection(s)
I20260925 10:03:03.146713 25190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:03.147653 25258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:03.150101 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a: Bootstrap starting.
I20260925 10:03:03.150665 25258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:03.150951 25258 log.cc:824] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:03.151695 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a: No bootstrap required, opened a new log
I20260925 10:03:03.153357 25258 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } }
I20260925 10:03:03.153577 25258 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:03.153606 25258 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea834d13b1db4e0e81f181046154945a, State: Initialized, Role: FOLLOWER
I20260925 10:03:03.153787 25258 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } }
I20260925 10:03:03.153851 25258 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:03.153903 25258 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:03.153954 25258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:03.154541 25258 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } }
I20260925 10:03:03.154609 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25190
I20260925 10:03:03.154660 25258 leader_election.cc:302] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea834d13b1db4e0e81f181046154945a; no voters:
I20260925 10:03:03.154698 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:03.154857 25258 leader_election.cc:288] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:03.154891 19643 external_mini_cluster.cc:1428] Setting key 548c0ded70e71aae04724898f3deae65
I20260925 10:03:03.154935 25261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:03.155128 25258 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:03.155326 25261 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [term 1 LEADER]: Becoming Leader. State: Replica: ea834d13b1db4e0e81f181046154945a, State: Running, Role: LEADER
I20260925 10:03:03.155479 25261 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } }
I20260925 10:03:03.155901 25263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea834d13b1db4e0e81f181046154945a. Latest consensus state: current_term: 1 leader_uuid: "ea834d13b1db4e0e81f181046154945a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } } }
I20260925 10:03:03.155982 25263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:03.156167 25262 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea834d13b1db4e0e81f181046154945a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea834d13b1db4e0e81f181046154945a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36925 } } }
I20260925 10:03:03.156258 25262 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:03.156574 25272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:03.157133 25272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:03.158748 25272 catalog_manager.cc:1347] Generated new cluster ID: 3ab433199d9a4caca4faec3817f75f2f
I20260925 10:03:03.158803 25272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:03.165385 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.155401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.19.46.254:36925)
0925 10:03:03.155745 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.155756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.155794 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:03:03.155934 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.155941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.156000 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.156862 (+ 862us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.157071 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.158161 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.159144 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.159566 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.159694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.162046 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.162050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.162051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.162063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.163787 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.164167 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.164170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.164170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.164272 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.164499 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.164500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.164501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.164754 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.164924 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.165029 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.165176 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260925 10:03:03.170008 25272 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:03.170534 25272 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:03.181286 25272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea834d13b1db4e0e81f181046154945a: Generated new TSK 0
I20260925 10:03:03.181555 25272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:03:03.212051 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:03.311826 25284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:03.312036 25284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:03.312067 25284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:03.313210 25284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:03.313254 25284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:03.313280 25284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:03.314837 25284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:03.315104 25284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:03.315346 25284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:03.320595 25289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.320595 25290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.320622 25292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.320698 25284 server_base.cc:1034] running on GCE node
I20260925 10:03:03.320998 25284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:03.321192 25284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:03.322328 25284 hybrid_clock.cc:648] HybridClock initialized: now 1790330583322291 us; error 52 us; skew 500 ppm
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.325047 25284 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:03:03.325641 25284 webserver.cc:460] Webserver started at http://127.19.46.193:45613/ using document root <none> and password file <none>
I20260925 10:03:03.325937 25284 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:03.326020 25284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:03.326157 25284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:03.327229 25284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "2d5edbf742f5407a822fa318bbb37e65"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "0ed56d3e9dc6b54b28e7335b8b6cecea"
server_key_iv: "12945958f134024d9c1b0010e1a95192"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.327646 25284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5edbf742f5407a822fa318bbb37e65"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "0ed56d3e9dc6b54b28e7335b8b6cecea"
server_key_iv: "12945958f134024d9c1b0010e1a95192"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.329003 25284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:03.329950 25299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.330127 25284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:03:03.330236 25284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "2d5edbf742f5407a822fa318bbb37e65"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "0ed56d3e9dc6b54b28e7335b8b6cecea"
server_key_iv: "12945958f134024d9c1b0010e1a95192"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.330313 25284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:03.348594 25284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:03.351116 25284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:03.351300 25284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:03.351558 25284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:03.351892 25284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:03.351948 25284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.351989 25284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:03.352032 25284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.358419 25284 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:43679
I20260925 10:03:03.358465 25412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:43679 every 8 connection(s)
I20260925 10:03:03.358825 25284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:03.367902 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25284
I20260925 10:03:03.368000 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:03.368206 19643 external_mini_cluster.cc:1428] Setting key 24ff4714b7ec9f6102cd1971a146c6c0
I20260925 10:03:03.368630 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.359576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:40089 (local address 127.19.46.254:36925)
0925 10:03:03.359668 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.359670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.359875 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:03:03.360860 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.360862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.360876 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.360933 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.361637 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.362160 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.363152 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.363373 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.363449 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.365765 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.365766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.365767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.365787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.367036 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.367430 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.367431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.367432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.367494 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.367796 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.367797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.367798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.367946 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.368056 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.368420 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.368528 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:03:03.368829 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.359534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36925 (local address 127.19.46.193:40089)
0925 10:03:03.359719 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:03.359780 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:03.360741 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:03.361006 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:03.361012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:03.361281 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:03.361544 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.361549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.362257 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.362258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.363057 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.363059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.363466 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.363466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.363595 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.364174 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:03.364183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:03.365658 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:03.367121 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.367127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.367130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.367353 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.367579 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.367580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.367580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.367709 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.368059 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:03.368066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:03.368239 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:03:03.368365 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 10:03:03.368564 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260925 10:03:03.369364 25413 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36925
I20260925 10:03:03.369527 25413 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:03.369717 25413 heartbeater.cc:511] Master 127.19.46.254:36925 requested a full tablet report, sending...
I20260925 10:03:03.370383 25222 ts_manager.cc:194] Registered new tserver with Master: 2d5edbf742f5407a822fa318bbb37e65 (127.19.46.193:43679)
I20260925 10:03:03.371045 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:40089
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:03:03.396317 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:03.480295 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:03.480573 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:03.480613 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:03.481829 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:03.481904 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:03.481950 25418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:03.483477 25418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:03.483755 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:03.483991 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:03.489560 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.489544 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.489614 25418 server_base.cc:1034] running on GCE node
W20260925 10:03:03.489544 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.490016 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:03.490257 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:03.491421 25418 hybrid_clock.cc:648] HybridClock initialized: now 1790330583491395 us; error 40 us; skew 500 ppm
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.494148 25418 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:03:03.494654 25418 webserver.cc:460] Webserver started at http://127.19.46.194:33699/ using document root <none> and password file <none>
I20260925 10:03:03.494904 25418 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:03.494984 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:03.495139 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:03.496218 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "d3e2bceaedfd472fbf3417411f8235f7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "f1e186d57125991768a176d3d00a9d51"
server_key_iv: "7871f6e0914807469d00be560cfe401f"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.496596 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "d3e2bceaedfd472fbf3417411f8235f7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "f1e186d57125991768a176d3d00a9d51"
server_key_iv: "7871f6e0914807469d00be560cfe401f"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.498085 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:03.498986 25433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.499218 25418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:03.499323 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "d3e2bceaedfd472fbf3417411f8235f7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "f1e186d57125991768a176d3d00a9d51"
server_key_iv: "7871f6e0914807469d00be560cfe401f"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.499418 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:03.520246 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:03.522682 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:03.522868 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:03.523118 25418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:03.523442 25418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:03.523507 25418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.523550 25418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:03.523578 25418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.530090 25418 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:40713
I20260925 10:03:03.530141 25546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:40713 every 8 connection(s)
I20260925 10:03:03.530478 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:03.531773 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25418
I20260925 10:03:03.531862 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:03.532109 19643 external_mini_cluster.cc:1428] Setting key dbcbacff5b0fb33d428b5cf9fa20b77b
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:03.541224 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.531200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:60547 (local address 127.19.46.254:36925)
0925 10:03:03.531268 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.531270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.531501 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:03:03.532525 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.532526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.532535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.532578 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.533520 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.534053 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.535140 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.535415 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.535476 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.538115 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.538117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.538117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.538127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.539781 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.540196 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.540198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.540198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.540259 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.540500 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.540501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.540501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.540657 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.540740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.541007 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.541116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:03:03.541437 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.531161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36925 (local address 127.19.46.194:60547)
0925 10:03:03.531335 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:03.531378 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:03.532413 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:03.532652 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:03.532656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:03.533086 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:03.533384 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.533390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.534131 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.534132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.535052 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.535054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.535508 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.535510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.535658 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.536246 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:03.536256 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:03.538010 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:03.539870 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.539877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.539879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.540112 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.540331 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.540332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.540332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.540429 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.540731 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:03.540735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:03.540868 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:03:03.540953 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:03:03.541125 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260925 10:03:03.541935 25547 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36925
I20260925 10:03:03.542057 25547 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:03.542243 25547 heartbeater.cc:511] Master 127.19.46.254:36925 requested a full tablet report, sending...
I20260925 10:03:03.542671 25222 ts_manager.cc:194] Registered new tserver with Master: d3e2bceaedfd472fbf3417411f8235f7 (127.19.46.194:40713)
I20260925 10:03:03.543078 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:60547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:03:03.561296 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:03.645829 25552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:03.646131 25552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:03.646159 25552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:03.647316 25552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:03.647363 25552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:03.647390 25552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:03.648849 25552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:03.649049 25552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:03.649230 25552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:03.654656 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.654721 25558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:03.654814 25557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:03.655112 25552 server_base.cc:1034] running on GCE node
I20260925 10:03:03.655372 25552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:03.655646 25552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:03.656898 25552 hybrid_clock.cc:648] HybridClock initialized: now 1790330583656875 us; error 40 us; skew 500 ppm
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.659739 25552 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:03:03.660200 25552 webserver.cc:460] Webserver started at http://127.19.46.195:42781/ using document root <none> and password file <none>
I20260925 10:03:03.660408 25552 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:03.660452 25552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:03.660555 25552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:03.661760 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "00f3946f4f5042119112b11ef1993fa7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "18696118a10ccf2acee5c4ced4240906"
server_key_iv: "aed1bd14df9b0b64716f4168fcd0c615"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.662109 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "00f3946f4f5042119112b11ef1993fa7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "18696118a10ccf2acee5c4ced4240906"
server_key_iv: "aed1bd14df9b0b64716f4168fcd0c615"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.663584 25552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:03:03.664384 25567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.664604 25552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:03.664731 25552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "00f3946f4f5042119112b11ef1993fa7"
format_stamp: "Formatted at 2026-09-25 10:03:03 on dist-test-slave-f9rl"
server_key: "18696118a10ccf2acee5c4ced4240906"
server_key_iv: "aed1bd14df9b0b64716f4168fcd0c615"
server_key_version: "encryptionkey@0"
I20260925 10:03:03.664821 25552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:03.678231 25552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:03.680900 25552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:03.681110 25552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:03.681460 25552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:03.681946 25552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:03.682024 25552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.682075 25552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:03.682104 25552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:03.688696 25552 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:42587
I20260925 10:03:03.688778 25680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:42587 every 8 connection(s)
I20260925 10:03:03.689141 25552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:03.696861 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25552
I20260925 10:03:03.697016 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:03.697211 19643 external_mini_cluster.cc:1428] Setting key 32434b328b26e500e4cfeee4fe0e232c
I20260925 10:03:03.699533 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.689960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:59641 (local address 127.19.46.254:36925)
0925 10:03:03.690006 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.690008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.690234 (+ 226us) server_negotiation.cc:407] Connection header received
0925 10:03:03.691213 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.691214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.691227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.691288 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.691935 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.692452 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.693508 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.693716 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.693804 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.696333 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.696335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.696335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.696343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.698085 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.698495 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.698496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.698497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.698556 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.698839 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.698840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.698841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.698976 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.699053 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.699345 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.699429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:03:03.699764 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.689929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36925 (local address 127.19.46.195:59641)
0925 10:03:03.690113 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:03.690149 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:03.691096 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:03.691359 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:03.691363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:03.691645 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:03.691839 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.691843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.692559 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.692561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.693373 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:03.693375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.693816 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.693817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:03.693952 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.694626 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:03.694635 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:03.696225 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:03.698147 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.698155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.698158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.698393 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.698631 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:03.698631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:03.698632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:03.698759 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:03.699044 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:03.699050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:03.699205 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:03:03.699322 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:03:03.699511 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260925 10:03:03.700269 25681 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36925
I20260925 10:03:03.700353 25681 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:03.700567 25681 heartbeater.cc:511] Master 127.19.46.254:36925 requested a full tablet report, sending...
I20260925 10:03:03.700982 25222 ts_manager.cc:194] Registered new tserver with Master: 00f3946f4f5042119112b11ef1993fa7 (127.19.46.195:42587)
I20260925 10:03:03.701361 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:59641
I20260925 10:03:03.709251 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/test-admin.keytab.
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.727136 19643 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 10:03:03.728856 25688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:03.728086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.19.46.193:43679)
0925 10:03:03.728252 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.728258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.728277 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:03:03.728305 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.728307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.728321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.728396 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.728736 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53014: BlockingRecv error: recv got EOF from 127.0.0.1:53014 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260925 10:03:03.728971 19643 init.cc:244] Error while opening credential cache '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc)
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.730000 19643 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.730958 19643 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:03:03.739565 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.731568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.19.46.193:43679)
0925 10:03:03.731651 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.731653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.731734 (+ 81us) server_negotiation.cc:407] Connection header received
0925 10:03:03.731821 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.731822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.731840 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.731907 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.732365 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.733118 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.733949 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.734295 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.734387 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.736660 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.736671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.736672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.736681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.738332 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.738728 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.738732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.738732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.738793 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.739002 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.739003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.739003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.739151 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.739310 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.739361 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.739442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:03:03.741402 25688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:03.740838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.19.46.193:43679)
0925 10:03:03.740878 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.740880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.740931 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:03:03.741010 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.741011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.741021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.741065 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.741350 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53026: BlockingRecv error: recv got EOF from 127.0.0.1:53026 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:03:03.741523 19643 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.742516 19643 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.743451 19643 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:03:03.750502 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.743968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.19.46.193:43679)
0925 10:03:03.744038 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.744040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.744100 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:03:03.744136 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.744137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.744146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.744193 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.744549 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.745022 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.745725 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.745915 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.745979 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.748033 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.748034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.748035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.748041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.749508 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.749815 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.749816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.749816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.749863 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.750037 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.750038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.750038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.750127 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.750198 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.750373 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.750434 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 10:03:03.752142 25688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:03.751635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.19.46.193:43679)
0925 10:03:03.751667 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.751669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.751724 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:03:03.751772 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.751772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.751779 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.751823 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.752076 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53038: BlockingRecv error: recv got EOF from 127.0.0.1:53038 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 10:03:03.752266 19643 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330556283750-19643-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.753183 19643 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:03.754055 19643 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:03:03.761546 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:03.754588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.19.46.193:43679)
0925 10:03:03.754647 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:03.754649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:03.754749 (+ 100us) server_negotiation.cc:407] Connection header received
0925 10:03:03.754826 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:03.754826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:03.754836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:03.754973 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:03.755347 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.755818 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.756630 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:03.756826 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:03.756911 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:03.759013 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:03.759014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:03.759014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:03.759022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:03.760407 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.760757 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.760758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.760759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.760816 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:03.761016 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:03.761017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:03.761018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:03.761122 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:03.761222 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:03.761328 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:03:03.761406 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:03:03.762259 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25284
I20260925 10:03:03.767443 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25418
I20260925 10:03:03.772611 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25552
I20260925 10:03:03.777190 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25190
2026-09-25T10:03:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1944 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25701](info): set up 2 sockets
Sep 25 10:03:03 dist-test-slave-f9rl krb5kdc[25701](info): commencing operation
krb5kdc: starting...
W20260925 10:03:04.839795 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:04 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab.
Entry for principal oryx/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab.
I20260925 10:03:04.972379 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44019
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab
--principal=oryx/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:44019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:05.058182 25714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:05.058473 25714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:05.058523 25714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:05.059859 25714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:05.059942 25714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:05.059969 25714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:05.059999 25714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:05.060034 25714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:05.061816 25714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:44019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab
--principal=oryx/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:05.062160 25714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:05.062412 25714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.067811 25720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.067802 25719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.067802 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.067831 25714 server_base.cc:1034] running on GCE node
I20260925 10:03:05.068387 25714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.068698 25714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.069867 25714 hybrid_clock.cc:648] HybridClock initialized: now 1790330585069831 us; error 49 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.072577 25714 init.cc:377] Logged in from keytab as oryx/127.19.46.254@KRBTEST.COM (short username oryx)
I20260925 10:03:05.073069 25714 webserver.cc:460] Webserver started at http://127.19.46.254:35245/ using document root <none> and password file <none>
I20260925 10:03:05.073313 25714 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.073385 25714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.073572 25714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:05.074705 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4ecca3724a51473ba1fda458a760c4a1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9f7b8e62a733993b9bea48f4d5e59585"
server_key_iv: "6b05a29c161e4187fa476dcbaac8aff2"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.075094 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4ecca3724a51473ba1fda458a760c4a1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9f7b8e62a733993b9bea48f4d5e59585"
server_key_iv: "6b05a29c161e4187fa476dcbaac8aff2"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.076586 25714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:05.077636 25729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.077893 25714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:05.078011 25714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4ecca3724a51473ba1fda458a760c4a1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9f7b8e62a733993b9bea48f4d5e59585"
server_key_iv: "6b05a29c161e4187fa476dcbaac8aff2"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.078131 25714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.090883 25714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.094493 25714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:05.094666 25714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.099786 25714 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:44019
I20260925 10:03:05.099822 25781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:44019 every 8 connection(s)
I20260925 10:03:05.100211 25714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:05.101257 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:05.104030 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Bootstrap starting.
I20260925 10:03:05.104732 25782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:05.105047 25782 log.cc:824] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:05.106045 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: No bootstrap required, opened a new log
I20260925 10:03:05.107776 25782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } }
I20260925 10:03:05.107995 25782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:05.108040 25782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecca3724a51473ba1fda458a760c4a1, State: Initialized, Role: FOLLOWER
I20260925 10:03:05.108073 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25714
I20260925 10:03:05.108172 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:05.108242 25782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } }
I20260925 10:03:05.108342 25782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:05.108377 25782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:05.108404 19643 external_mini_cluster.cc:1428] Setting key b551a4488d19b311b1c062deffcfbfaf
I20260925 10:03:05.108443 25782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:05.109647 25782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } }
I20260925 10:03:05.109776 25782 leader_election.cc:302] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecca3724a51473ba1fda458a760c4a1; no voters:
I20260925 10:03:05.109946 25782 leader_election.cc:288] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:05.110023 25787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:05.110167 25787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecca3724a51473ba1fda458a760c4a1, State: Running, Role: LEADER
I20260925 10:03:05.110234 25782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:05.110263 25787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } }
I20260925 10:03:05.110632 25789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } }
I20260925 10:03:05.110625 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecca3724a51473ba1fda458a760c4a1. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } }
I20260925 10:03:05.110715 25789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:05.110728 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:05.110965 25794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:05.111498 25794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:05.113106 25794 catalog_manager.cc:1347] Generated new cluster ID: 78aaadd1e3284d56809c19d848002c2c
I20260925 10:03:05.113207 25794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:05.119743 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.109010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.19.46.254:44019)
0925 10:03:05.109200 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.109216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.109247 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:03:05.109346 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.109352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.109395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.109574 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.110423 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.111889 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.112955 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.113631 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.113765 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.116151 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.116155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.116156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.116169 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.118022 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.118420 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.118426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.118426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.118533 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.118770 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.118771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.118772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.119059 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.119232 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.119348 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.119517 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260925 10:03:05.138403 25794 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:05.139184 25794 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:05.155980 25794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Generated new TSK 0
I20260925 10:03:05.156215 25794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
I20260925 10:03:05.184055 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:05.267884 25808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:05.268110 25808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:05.268136 25808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:05.269335 25808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:05.269380 25808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:05.269407 25808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:05.270967 25808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:05.271242 25808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:05.271490 25808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.276770 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.276794 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.276849 25808 server_base.cc:1034] running on GCE node
W20260925 10:03:05.276847 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.277245 25808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.277575 25808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.278748 25808 hybrid_clock.cc:648] HybridClock initialized: now 1790330585278719 us; error 37 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.281620 25808 init.cc:377] Logged in from keytab as oryx/127.19.46.193@KRBTEST.COM (short username oryx)
I20260925 10:03:05.282166 25808 webserver.cc:460] Webserver started at http://127.19.46.193:41567/ using document root <none> and password file <none>
I20260925 10:03:05.282438 25808 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.282511 25808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.282670 25808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:05.283779 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "5d0f9049117147d1ab05ffc5627e0df1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "3ad3c58e05d5872f9deaccc7857697f4"
server_key_iv: "1955c88943d14b29672b9ba29892d481"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.284147 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "5d0f9049117147d1ab05ffc5627e0df1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "3ad3c58e05d5872f9deaccc7857697f4"
server_key_iv: "1955c88943d14b29672b9ba29892d481"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.285619 25808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:05.286465 25823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.286655 25808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:05.286769 25808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "5d0f9049117147d1ab05ffc5627e0df1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "3ad3c58e05d5872f9deaccc7857697f4"
server_key_iv: "1955c88943d14b29672b9ba29892d481"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.286866 25808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.303932 25808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.306532 25808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:05.306720 25808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.306977 25808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:05.307313 25808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:05.307372 25808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.307420 25808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:05.307473 25808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.314435 25808 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:37819
I20260925 10:03:05.314486 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:37819 every 8 connection(s)
I20260925 10:03:05.314849 25808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:05.319651 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25808
I20260925 10:03:05.319777 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:05.320026 19643 external_mini_cluster.cc:1428] Setting key 10f9efa42fffad05b7c0e6edaf5cbdde
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:05.325575 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.315587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:56683)
0925 10:03:05.315752 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.315803 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.316400 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.316673 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.316678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.317337 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.317628 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.317633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.318423 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.318424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.319321 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.319323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.319896 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.319898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.320026 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.320602 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.320611 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.322325 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.324008 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.324015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.324017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.324244 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.324531 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.324531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.324532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.324629 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.324990 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.324995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.325116 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.325202 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.325324 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260925 10:03:05.325584 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.315618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56683 (local address 127.19.46.254:44019)
0925 10:03:05.315689 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.315692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.315926 (+ 234us) server_negotiation.cc:407] Connection header received
0925 10:03:05.316519 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.316520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.316538 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.316600 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.317739 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.318297 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.319433 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.319783 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.319858 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.322454 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.322456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.322456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.322463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.323902 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.324368 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.324370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.324370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.324447 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.324708 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.324709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.324710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.324913 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.324997 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.325276 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.325394 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:03:05.326297 25937 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:05.326388 25937 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:05.326573 25937 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:05.327140 25738 ts_manager.cc:194] Registered new tserver with Master: 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819)
I20260925 10:03:05.327805 25738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:56683
WARNING: no policy specified for oryx/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
I20260925 10:03:05.347918 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:05.431191 25942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:05.431444 25942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:05.431486 25942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:05.432657 25942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:05.432726 25942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:05.432773 25942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:05.434312 25942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:05.434600 25942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:05.434841 25942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.439838 25947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.440006 25942 server_base.cc:1034] running on GCE node
W20260925 10:03:05.439843 25948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.439869 25950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.440325 25942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.440585 25942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.441746 25942 hybrid_clock.cc:648] HybridClock initialized: now 1790330585441735 us; error 46 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.444293 25942 init.cc:377] Logged in from keytab as oryx/127.19.46.194@KRBTEST.COM (short username oryx)
I20260925 10:03:05.444758 25942 webserver.cc:460] Webserver started at http://127.19.46.194:42157/ using document root <none> and password file <none>
I20260925 10:03:05.445001 25942 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.445071 25942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.445220 25942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:05.446355 25942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "813640b15f584c4e94b1fdc9d52ee31c"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9fcfbca3c81193655a185704a8c62c5e"
server_key_iv: "2410a0f1d7c7c25f142131fa1f6a58bf"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.446722 25942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "813640b15f584c4e94b1fdc9d52ee31c"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9fcfbca3c81193655a185704a8c62c5e"
server_key_iv: "2410a0f1d7c7c25f142131fa1f6a58bf"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.448690 25942 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:03:05.449530 25957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.449700 25942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:05.449798 25942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "813640b15f584c4e94b1fdc9d52ee31c"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9fcfbca3c81193655a185704a8c62c5e"
server_key_iv: "2410a0f1d7c7c25f142131fa1f6a58bf"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.449921 25942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.461328 25942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.463646 25942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:05.463842 25942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.464090 25942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:05.464429 25942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:05.464493 25942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.464540 25942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:05.464563 25942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.470682 25942 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:33811
I20260925 10:03:05.470736 26070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:33811 every 8 connection(s)
I20260925 10:03:05.471095 25942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:05.473210 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 25942
I20260925 10:03:05.473352 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:05.473587 19643 external_mini_cluster.cc:1428] Setting key b5e59689e23bb94f70327d2e82ec0674
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:05.483145 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.472031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:54717 (local address 127.19.46.254:44019)
0925 10:03:05.472130 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.472132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.472368 (+ 236us) server_negotiation.cc:407] Connection header received
0925 10:03:05.473040 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.473042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.473058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.473144 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.474460 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.475252 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.476856 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.477088 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.477164 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.479636 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.479639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.479639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.479656 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.481567 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.481998 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.482000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.482000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.482075 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.482343 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.482345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.482345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.482548 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.482627 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.482906 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.483003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:03:05.483179 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.472026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:54717)
0925 10:03:05.472223 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.472269 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.472829 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.473233 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.473237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.474094 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.474374 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.474380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.475382 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.475383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.476713 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.476717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.477183 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.477184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.477319 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.477974 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.477983 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.479521 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.481659 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.481666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.481668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.481900 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.482155 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.482156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.482157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.482262 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.482621 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.482626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.482756 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.482845 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.483000 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:03:05.483963 26071 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:05.484097 26071 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:05.484297 26071 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:05.484752 25738 ts_manager.cc:194] Registered new tserver with Master: 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811)
I20260925 10:03:05.485167 25738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:54717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
I20260925 10:03:05.504987 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:05.590732 26076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:05.590965 26076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:05.590992 26076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:05.592144 26076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:05.592183 26076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:05.592211 26076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:05.593780 26076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:44019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:05.593994 26076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:05.594190 26076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.599720 26081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.599701 26082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.599721 26084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.599746 26076 server_base.cc:1034] running on GCE node
I20260925 10:03:05.600298 26076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.600567 26076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.601745 26076 hybrid_clock.cc:648] HybridClock initialized: now 1790330585601736 us; error 36 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.604952 26076 init.cc:377] Logged in from keytab as oryx/127.19.46.195@KRBTEST.COM (short username oryx)
I20260925 10:03:05.605556 26076 webserver.cc:460] Webserver started at http://127.19.46.195:43833/ using document root <none> and password file <none>
I20260925 10:03:05.605860 26076 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.605929 26076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.606067 26076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:05.607379 26076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "fbd72b40b5ed4a15975b5e46061b0747"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "ed73db7369e1549c5e76a8bfc530c3a9"
server_key_iv: "65ca7b3ef8ca3fc4cffc595f4790488f"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.607776 26076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "fbd72b40b5ed4a15975b5e46061b0747"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "ed73db7369e1549c5e76a8bfc530c3a9"
server_key_iv: "65ca7b3ef8ca3fc4cffc595f4790488f"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.609225 26076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:05.610296 26091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.610530 26076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:05.610634 26076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "fbd72b40b5ed4a15975b5e46061b0747"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "ed73db7369e1549c5e76a8bfc530c3a9"
server_key_iv: "65ca7b3ef8ca3fc4cffc595f4790488f"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.610710 26076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.624224 26076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.626680 26076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:05.626837 26076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.627105 26076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:05.627427 26076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:05.627461 26076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.627487 26076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:05.627553 26076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.633503 26076 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:40869
I20260925 10:03:05.633589 26204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:40869 every 8 connection(s)
I20260925 10:03:05.633909 26076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:05.640532 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26076
I20260925 10:03:05.640635 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:05.640837 19643 external_mini_cluster.cc:1428] Setting key c759f15943cb7eb6745c8295ef1ae983
I20260925 10:03:05.643746 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.634686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:38057 (local address 127.19.46.254:44019)
0925 10:03:05.634732 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.634734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.634942 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:03:05.635453 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.635454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.635466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.635542 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.636531 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.637021 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.638201 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.638387 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.638452 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.640860 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.640863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.640863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.640872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.642495 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.642878 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.642879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.642879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.642928 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.643155 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.643156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.643156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.643263 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.643315 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.643618 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.643680 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:03:05.644047 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.634642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:38057)
0925 10:03:05.634810 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.634855 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.635355 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.635621 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.635625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.636241 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.636441 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.636445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.637130 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.637131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.638108 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.638111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.638490 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.638492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.638609 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.639226 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.639235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.640768 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.642572 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.642578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.642579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.642803 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.642986 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.642987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.642987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.643087 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.643347 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.643354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.643491 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.643592 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.643737 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260925 10:03:05.644557 26205 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:05.644668 26205 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:05.644850 26205 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:05.645303 25738 ts_manager.cc:194] Registered new tserver with Master: fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:05.645725 25738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:38057
I20260925 10:03:05.653384 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.253.keytab.
Entry for principal oryx/127.19.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.253.keytab.
I20260925 10:03:05.681644 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:42999
--webserver_interface=127.19.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.253.keytab
--principal=oryx/127.19.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:44019,127.19.46.253:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:05.767335 26210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:05.767589 26210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:05.767614 26210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:05.768764 26210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:05.768807 26210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:05.768822 26210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:05.768834 26210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:05.768846 26210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:05.770344 26210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:44019,127.19.46.253:42999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.253.keytab
--principal=oryx/127.19.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
W20260925 10:03:05.770452 26210 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.46.254:44019,127.19.46.253:42999'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:03:05.770571 26210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:05.770752 26210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.775743 26218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.775743 26215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.775856 26216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.776126 26210 server_base.cc:1034] running on GCE node
I20260925 10:03:05.776460 26210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.776772 26210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.777984 26210 hybrid_clock.cc:648] HybridClock initialized: now 1790330585777957 us; error 36 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.781119 26210 init.cc:377] Logged in from keytab as oryx/127.19.46.253@KRBTEST.COM (short username oryx)
I20260925 10:03:05.781730 26210 webserver.cc:460] Webserver started at http://127.19.46.253:38239/ using document root <none> and password file <none>
I20260925 10:03:05.781977 26210 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.782045 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.782181 26210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:05.783263 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/instance:
uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "1a22372d583482272cd07fb6e73d46c8"
server_key_iv: "e99802749f57ee2b2556f85999b348af"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.783674 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal/instance:
uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "1a22372d583482272cd07fb6e73d46c8"
server_key_iv: "e99802749f57ee2b2556f85999b348af"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.785090 26210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:05.786003 26225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.786207 26210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:03:05.786314 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "1a22372d583482272cd07fb6e73d46c8"
server_key_iv: "e99802749f57ee2b2556f85999b348af"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.786401 26210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.819054 26210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.821410 26210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:05.821611 26210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.825398 26210 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.253:42999
I20260925 10:03:05.825523 26277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.253:42999 every 8 connection(s)
I20260925 10:03:05.825908 26210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
I20260925 10:03:05.826952 26278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:05.827253 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26210
I20260925 10:03:05.829746 26278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.253@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:05.839408 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.827976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.19.46.253:42999)
0925 10:03:05.828142 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.828161 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.828195 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:03:05.828989 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.828999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.829065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.829224 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.829964 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.831193 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.831946 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.832559 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.832664 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.835587 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.835593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.835593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.835599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.837665 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.838089 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.838092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.838093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.838172 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.838372 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.838373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.838373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.838720 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.838872 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.838995 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.839150 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260925 10:03:05.839633 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.830407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40110)
0925 10:03:05.830630 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.830657 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.830713 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.830854 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.830856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.831122 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.831187 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.831192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.832773 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.832774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.833733 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.833737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.834134 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.834135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.834207 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.834728 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.834737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.836243 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.838374 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.838380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.838382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.838657 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.838888 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.838889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.838890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.838981 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.839287 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.839292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.839340 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.839426 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.839541 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
I20260925 10:03:05.839998 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.830483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.19.46.254:44019)
0925 10:03:05.830555 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.830557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.830716 (+ 159us) server_negotiation.cc:407] Connection header received
0925 10:03:05.830752 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.830753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.830766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.830832 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.831967 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.832668 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.833842 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.834043 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.834101 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.836352 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.836354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.836355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.836362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.838179 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.838733 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.838734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.838734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.838815 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.839056 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.839057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.839057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.839236 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.839290 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.839451 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.839571 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:03:05.840538 26278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.253@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
I20260925 10:03:05.848094 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.840877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.19.46.253:42999)
0925 10:03:05.840922 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.840924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.840931 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:03:05.841094 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.841095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.841104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.841163 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.841744 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.842223 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.842884 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.843075 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.843133 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.845298 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.845300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.845300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.845307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.846822 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.847221 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.847222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.847222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.847275 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.847475 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.847476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.847476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.847605 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.847679 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.847814 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.847900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:03:05.848187 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.840826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35050)
0925 10:03:05.840877 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.840889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.840952 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.841216 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.841218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.841338 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.841612 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.841615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.842330 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.842331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.842806 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.842808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.843174 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.843175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.843235 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.843732 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.843739 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.845176 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.846943 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.846944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.846944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.847129 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.847349 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.847349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.847350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.847406 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.847669 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.847670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.847707 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.847792 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.847949 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":68}
I20260925 10:03:05.849910 26278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Bootstrap starting.
I20260925 10:03:05.850747 26278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:05.851001 26278 log.cc:824] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:05.851786 26278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: No bootstrap required, opened a new log
I20260925 10:03:05.853310 26278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } }
I20260925 10:03:05.853566 26278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:05.853596 26278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49895b4e59ff41f9aed9a33380c4b5ae, State: Initialized, Role: FOLLOWER
I20260925 10:03:05.853730 26278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } }
I20260925 10:03:05.853991 26284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } } }
I20260925 10:03:05.854064 26278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:05.854084 26284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:03:05.855629 26295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:03:05.855700 26295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:03:05.857571 26210 master_runner.cc:428] Detected that this master 49895b4e59ff41f9aed9a33380c4b5ae is joining an existing cluster
I20260925 10:03:05.857630 26210 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.46.253:42999
I20260925 10:03:05.865971 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.858744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.19.46.254:44019)
0925 10:03:05.858793 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.858795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.858908 (+ 113us) server_negotiation.cc:407] Connection header received
0925 10:03:05.858987 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.858988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.858998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.859048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.859476 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.860158 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861005 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.861315 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861388 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.862579 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.862581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.862582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.862591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.864593 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865056 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.865057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.865058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.865131 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865377 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.865378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.865379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.865560 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.865621 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.865791 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.865866 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:03:05.865957 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.859207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35064)
0925 10:03:05.859345 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.859358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.859419 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.859549 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.859549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.859740 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.859816 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.859818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.860710 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.860711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.861327 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.861329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.861943 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.862040 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.862517 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.862521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.862791 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.864615 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.864616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.864617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.864965 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.865179 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.865180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.865235 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.865538 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.865539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.865592 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.865684 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.865821 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260925 10:03:05.866071 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.858755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.19.46.253:42999)
0925 10:03:05.858793 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.858795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.859432 (+ 637us) server_negotiation.cc:407] Connection header received
0925 10:03:05.859473 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.859474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.859484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.859547 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.859903 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.860610 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861431 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.861725 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861766 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.862875 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.862876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.862877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.862884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.864406 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865055 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.865056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.865056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.865103 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865307 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.865308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.865308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.865470 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.865534 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.865707 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.865931 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":19}
I20260925 10:03:05.866207 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.858705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40120)
0925 10:03:05.858852 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.858862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.858924 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.859051 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.859052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.859248 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.859379 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.859383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.860231 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.860232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.860913 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.860915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.861413 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.861414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.861518 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.862064 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.862069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.862473 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.864680 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.864681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.864681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.864973 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.865225 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.865226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.865226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.865311 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.865611 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.865612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.865649 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.865759 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.865971 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.254@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
I20260925 10:03:05.879501 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.869330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.19.46.253:42999)
0925 10:03:05.869372 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.869374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.869595 (+ 221us) server_negotiation.cc:407] Connection header received
0925 10:03:05.869697 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.869698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.869708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.869758 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.870624 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.871228 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.872376 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.872655 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.872702 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.875850 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.875852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.875852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.875861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.877901 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.878476 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.878477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.878478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.878544 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.878762 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.878762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.878763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.878913 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.878962 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.879241 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.879334 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:03:05.880088 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.869254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35074)
0925 10:03:05.869483 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.869530 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.869623 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.869761 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.869764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.870331 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.870438 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.870445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.871360 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.871362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.872268 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.872272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.872793 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.872794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.872902 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.873646 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.873652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.875719 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.878023 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.878034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.878037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.878356 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.878626 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.878626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.878627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.878694 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.879018 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.879027 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.879093 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.879875 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.880006 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260925 10:03:05.880889 25738 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.46.253:42999: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
member_type: NON_VOTER
last_known_addr {
host: "127.19.46.253"
port: 42999
}
attrs {
promote: true
}
}
dest_uuid: "4ecca3724a51473ba1fda458a760c4a1"
cas_config_opid_index: -1
I20260925 10:03:05.881250 25738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } }
I20260925 10:03:05.881793 26305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } }
I20260925 10:03:05.882202 26252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:05.882843 26305 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.46.253:42999
I20260925 10:03:05.883072 26210 master.cc:537] Master@127.19.46.253:42999 shutting down...
I20260925 10:03:05.883108 26304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } }
I20260925 10:03:05.883188 26304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:05.883322 26252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 10:03:05.883701 26304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:03:05.884497 25733 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 10:03:05.884956 25733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 -> Peer 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253:42999): Couldn't send request to peer 49895b4e59ff41f9aed9a33380c4b5ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:03:05.886257 26210 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:05.886359 26210 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:05.886422 26210 tablet_replica.cc:330] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: stopping tablet replica
I20260925 10:03:05.899196 26210 master.cc:552] Master@127.19.46.253:42999 shutdown complete.
I20260925 10:03:05.899256 26210 master_runner.cc:305] Clearing existing system tablet
I20260925 10:03:05.899400 26210 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.899452 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.900127 26210 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:05.900636 26309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.900759 26210 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.900817 26210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.900839 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "1a22372d583482272cd07fb6e73d46c8"
server_key_iv: "e99802749f57ee2b2556f85999b348af"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.901623 26210 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:05.903304 26210 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:03:05.903364 26210 log.cc:1196] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:03:05.903569 26210 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Deleting consensus metadata
I20260925 10:03:05.903697 26210 master_runner.cc:315] Copying system tablet from 127.19.46.254:44019
I20260925 10:03:05.904385 26210 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: tablet copy: Beginning tablet copy session from remote peer at address 127.19.46.254:44019
I20260925 10:03:05.910908 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.904820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.19.46.254:44019)
0925 10:03:05.904873 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.904876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.904935 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:03:05.905046 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.905047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.905058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.905110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.905474 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.905986 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.906733 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.906940 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.907019 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.908089 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.908090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.908091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.908098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.909592 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.910028 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.910029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.910029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.910080 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.910298 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.910299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.910299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.910453 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.910503 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.910677 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.910769 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:03:05.910923 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.904738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40122)
0925 10:03:05.904861 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:05.904875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:05.904954 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:05.905114 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:05.905115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:05.905299 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:05.905357 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.905359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.906086 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.906087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.906646 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:05.906648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.907019 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.907019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:05.907085 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.907653 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:05.907658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:05.907963 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:05.909694 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.909696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.909696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.909947 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.910148 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:05.910149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:05.910149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:05.910223 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:05.910517 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:05.910518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:05.910573 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 10:03:05.910662 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:03:05.910797 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260925 10:03:05.911533 25766 tablet_copy_service.cc:140] P 4ecca3724a51473ba1fda458a760c4a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 49895b4e59ff41f9aed9a33380c4b5ae ({username='oryx', principal='oryx/127.19.46.253@KRBTEST.COM'} at 127.0.0.1:40122)
I20260925 10:03:05.911631 25766 tablet_copy_service.cc:161] P 4ecca3724a51473ba1fda458a760c4a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 49895b4e59ff41f9aed9a33380c4b5ae at {username='oryx', principal='oryx/127.19.46.253@KRBTEST.COM'} at 127.0.0.1:40122: session id = 49895b4e59ff41f9aed9a33380c4b5ae-00000000000000000000000000000000
I20260925 10:03:05.912271 25766 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:03:05.912927 26210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:03:05.914121 26210 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: tablet copy: Starting download of 0 data blocks...
I20260925 10:03:05.914271 26210 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: tablet copy: Starting download of 1 WAL segments...
I20260925 10:03:05.915146 26210 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:03:05.915959 25766 tablet_copy_service.cc:342] P 4ecca3724a51473ba1fda458a760c4a1: Request end of tablet copy session 49895b4e59ff41f9aed9a33380c4b5ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.46.253@KRBTEST.COM'} at 127.0.0.1:40122
I20260925 10:03:05.916020 25766 tablet_copy_service.cc:433] P 4ecca3724a51473ba1fda458a760c4a1: ending tablet copy session 49895b4e59ff41f9aed9a33380c4b5ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 49895b4e59ff41f9aed9a33380c4b5ae
I20260925 10:03:05.918574 26210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:05.919920 26316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.920109 26317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:05.920261 26319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:05.920305 26210 server_base.cc:1034] running on GCE node
I20260925 10:03:05.920392 26210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:05.920545 26210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:05.921665 26210 hybrid_clock.cc:648] HybridClock initialized: now 1790330585921651 us; error 29 us; skew 500 ppm
Sep 25 10:03:05 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:05.923332 26210 init.cc:377] Logged in from keytab as oryx/127.19.46.253@KRBTEST.COM (short username oryx)
I20260925 10:03:05.923638 26210 webserver.cc:460] Webserver started at http://127.19.46.253:35383/ using document root <none> and password file <none>
I20260925 10:03:05.923761 26210 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:05.923820 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:05.924497 26210 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:05.925037 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:05.925160 26210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:05.925207 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "1a22372d583482272cd07fb6e73d46c8"
server_key_iv: "e99802749f57ee2b2556f85999b348af"
server_key_version: "encryptionkey@0"
I20260925 10:03:05.925271 26210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:05.935635 26210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:05.937201 26210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:05.940383 26210 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.253:42999
I20260925 10:03:05.940795 26378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.253:42999 every 8 connection(s)
I20260925 10:03:05.940915 26210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
I20260925 10:03:05.942003 26379 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:03:05.942422 26379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Bootstrap starting.
I20260925 10:03:05.945055 26379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:03:05.945312 26379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: Bootstrap complete.
I20260925 10:03:05.945571 26379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } }
I20260925 10:03:05.945669 26379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49895b4e59ff41f9aed9a33380c4b5ae, State: Initialized, Role: LEARNER
I20260925 10:03:05.945711 26379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } }
I20260925 10:03:05.945914 26379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:05.945955 26383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } }
I20260925 10:03:05.946034 26383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: LEARNER
I20260925 10:03:05.947409 26394 catalog_manager.cc:1259] Loaded cluster ID: 78aaadd1e3284d56809c19d848002c2c
I20260925 10:03:05.947499 26394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: loading cluster ID for follower catalog manager: success
I20260925 10:03:05.948932 26394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: acquiring CA information for follower catalog manager: success
I20260925 10:03:05.949352 26394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:03:05.950157 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:05.941172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.19.46.253:42999)
0925 10:03:05.941340 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:05.941343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:05.941407 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:03:05.941675 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:05.941676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:05.941687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:05.941765 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:05.942581 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.943741 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.944754 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:05.945122 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:05.945517 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:05.946810 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:05.946811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:05.946811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:05.946819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:05.948808 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.949289 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.949290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.949291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.949351 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:05.949638 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:05.949638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:05.949639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:05.949738 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:05.949844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:05.949962 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:03:05.950054 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.19.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.252.keytab.
Entry for principal oryx/127.19.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.252.keytab.
I20260925 10:03:05.978750 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:44641
--webserver_interface=127.19.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.252.keytab
--principal=oryx/127.19.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:06.066793 26397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:06.067032 26397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:06.067056 26397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:06.068306 26397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:06.068356 26397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:06.068372 26397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:06.068385 26397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:06.068396 26397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:06.070096 26397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.252.keytab
--principal=oryx/127.19.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:06.070369 26397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:06.070633 26397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:06.076337 26405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.076318 26403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.076305 26402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.076344 26397 server_base.cc:1034] running on GCE node
I20260925 10:03:06.076756 26397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:06.077153 26397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:06.078332 26397 hybrid_clock.cc:648] HybridClock initialized: now 1790330586078307 us; error 48 us; skew 500 ppm
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:06.081058 26397 init.cc:377] Logged in from keytab as oryx/127.19.46.252@KRBTEST.COM (short username oryx)
I20260925 10:03:06.081650 26397 webserver.cc:460] Webserver started at http://127.19.46.252:36993/ using document root <none> and password file <none>
I20260925 10:03:06.081852 26397 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.081913 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.082019 26397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:06.083148 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/instance:
uuid: "6c195dab1cf947f3b3bfd9141c796c04"
format_stamp: "Formatted at 2026-09-25 10:03:06 on dist-test-slave-f9rl"
server_key: "8df5e7af5744f7cd734b2f6341bb29e0"
server_key_iv: "06ef7ab133321482c922b2cd04986cc0"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.083455 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal/instance:
uuid: "6c195dab1cf947f3b3bfd9141c796c04"
format_stamp: "Formatted at 2026-09-25 10:03:06 on dist-test-slave-f9rl"
server_key: "8df5e7af5744f7cd734b2f6341bb29e0"
server_key_iv: "06ef7ab133321482c922b2cd04986cc0"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.084821 26397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:06.085678 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.085952 26397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:06.086037 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "6c195dab1cf947f3b3bfd9141c796c04"
format_stamp: "Formatted at 2026-09-25 10:03:06 on dist-test-slave-f9rl"
server_key: "8df5e7af5744f7cd734b2f6341bb29e0"
server_key_iv: "06ef7ab133321482c922b2cd04986cc0"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.086135 26397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.101737 26397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:06.104043 26397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:06.104178 26397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:06.108642 26397 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.252:44641
I20260925 10:03:06.108803 26464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.252:44641 every 8 connection(s)
I20260925 10:03:06.109138 26397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
I20260925 10:03:06.110316 26465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:06.112123 26465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:03:06.116247 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26397
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.252@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.123420 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.113097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.19.46.254:44019)
0925 10:03:06.113186 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.113189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.113201 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:06.113888 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.113890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.113904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.113964 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.114705 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.115204 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.116434 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.116974 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.117053 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.119622 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.119624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.119624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.119632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.121288 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.121786 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.121787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.121787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.121873 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.122143 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.122144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.122145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.122270 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.122347 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.123222 (+ 875us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.123325 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:03:06.123625 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.112837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40134)
0925 10:03:06.113022 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.113068 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.113770 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.114048 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.114053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.114384 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.114587 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.114592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.115305 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.115307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.116228 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.116231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.117083 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.117085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.117206 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.117784 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.117796 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.119521 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.121399 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.121408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.121411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.121693 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.121942 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.121943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.121944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.122066 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.122876 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.122882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.123068 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.123180 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.123361 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260925 10:03:06.124315 26465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:03:06.127187 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.116920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.19.46.252:44641)
0925 10:03:06.117146 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.117151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.117171 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:03:06.117200 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.117202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.117218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.117278 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.118107 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.119050 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.119839 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.120283 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.120371 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.123340 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.123348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.123349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.123364 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.125657 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.126126 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.126129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.126129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.126229 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.126520 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.126521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.126522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.126695 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.126861 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.126982 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.127073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.252@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
I20260925 10:03:06.132700 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.124646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35090)
0925 10:03:06.124699 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.124711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.124752 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.125023 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.125023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.125162 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.125208 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.125211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.126196 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.126197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.126971 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.126973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.127538 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.127539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.127601 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.128039 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.128043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.129623 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.131452 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.131454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.131455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.131649 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.131915 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.131916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.131916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.131986 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.132311 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.132312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.132365 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.132460 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.132582 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:03:06.132701 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.124742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.19.46.253:42999)
0925 10:03:06.124796 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.124798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.124808 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:06.124932 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.124934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.124946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.125008 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.125330 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.126107 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.127069 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.127378 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.127440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.129731 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.129733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.129733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.129742 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.131314 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.131752 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.131753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.131754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.131834 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.132067 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.132068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.132068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.132211 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.132310 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.132492 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.132605 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:03:06.133378 26465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.252@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.141390 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.133744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42028)
0925 10:03:06.133822 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.133834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.133881 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.134150 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.134152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.134334 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.134400 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.134402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.135256 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.135258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.135834 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.135836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.136341 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.136342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.136390 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.136823 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.136828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.138548 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.140229 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.140230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.140231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.140416 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.140648 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.140649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.140649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.140717 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.141056 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.141057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.141105 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.141194 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.141285 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:03:06.141502 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.133865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.19.46.252:44641)
0925 10:03:06.133936 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.133938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.133946 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:06.133967 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.133968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.133985 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.134044 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.134535 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.135138 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.135997 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.136236 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.136308 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.138640 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.138641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.138642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.138649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.140136 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.140496 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.140497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.140497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.140550 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.140819 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.140820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.140821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.140961 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.141019 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.141236 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.141334 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":51}
I20260925 10:03:06.143424 26465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Bootstrap starting.
I20260925 10:03:06.144393 26465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:06.144709 26465 log.cc:824] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:06.145856 26465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: No bootstrap required, opened a new log
I20260925 10:03:06.147696 26465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } }
I20260925 10:03:06.147908 26465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:06.147964 26465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c195dab1cf947f3b3bfd9141c796c04, State: Initialized, Role: FOLLOWER
I20260925 10:03:06.148138 26465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } }
I20260925 10:03:06.148437 26470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } } }
I20260925 10:03:06.148511 26465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:06.148533 26470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:03:06.150321 26481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:03:06.150385 26481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:03:06.153632 26397 master_runner.cc:428] Detected that this master 6c195dab1cf947f3b3bfd9141c796c04 is joining an existing cluster
I20260925 10:03:06.153703 26397 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.46.252:44641
I20260925 10:03:06.162664 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.19.46.253:42999)
0925 10:03:06.155185 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.155187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.155613 (+ 426us) server_negotiation.cc:407] Connection header received
0925 10:03:06.155705 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.155706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.155717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.155747 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.156182 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.156948 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158147 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158460 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158509 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.159777 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.159779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.159779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.159785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.161316 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.161781 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.161782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.161782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.161843 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.162111 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.162112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.162112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.162257 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.162337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.162508 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.162586 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:03:06.162709 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35094)
0925 10:03:06.155559 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.155569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.155634 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.155808 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.155809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.156028 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.156092 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.156094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.157050 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.157051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158043 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.158046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158612 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158690 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.159186 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.159191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.159670 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.161411 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.161412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.161413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.161688 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.161921 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.161922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.161923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.162001 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.162346 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.162347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.162398 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.162474 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.162588 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":79,"thread_start_us":228,"threads_started":1}
I20260925 10:03:06.178720 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25808
I20260925 10:03:06.185343 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:37819
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:44019
--webserver_port=41567
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:03:06.186883 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40148)
0925 10:03:06.155380 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.155393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.155469 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.155658 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.155659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.155940 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.156009 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.156011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.157032 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.157033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158039 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.158041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158626 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158728 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.182223 (+ 23495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.182230 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.182763 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.185678 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.185679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.185680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.185894 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.186116 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.186117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.186117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.186179 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.186475 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.186476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.186527 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.186583 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.186717 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":44,"thread_start_us":236,"threads_started":1}
I20260925 10:03:06.187052 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.19.46.252:44641)
0925 10:03:06.155602 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.155603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.156189 (+ 586us) server_negotiation.cc:407] Connection header received
0925 10:03:06.156264 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.156265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.156272 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.156297 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.156656 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.157328 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158288 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158561 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158599 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.183793 (+ 25194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.183795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.183795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.183802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.185593 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.186224 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.186226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.186226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.186290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.186484 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.186484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.186485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.186614 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.186656 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.186864 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.186948 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:03:06.187338 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.19.46.254:44019)
0925 10:03:06.155226 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.155233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.155478 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:03:06.155522 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.155523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.155537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.155601 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.156102 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.156946 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158126 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158439 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158526 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.182914 (+ 24388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.182917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.182918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.182929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.185369 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.185989 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.185991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.185991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.186051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.186237 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.186238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.186238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.186393 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.186449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.187123 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.187221 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:03:06.188095 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.155649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42032)
0925 10:03:06.156143 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.156152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.156208 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.156344 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.156344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.156530 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.156581 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.156583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.157406 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.157406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158217 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.158219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.158661 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.158662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.158718 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.183108 (+ 24390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.183117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.183682 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.185889 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.185890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.185891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.186131 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.186360 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.186361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.186361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.186416 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.186699 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.186699 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.186754 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.187851 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.187972 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":69,"threads_started":1}
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330585, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.254@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.198922 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.189869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42040)
0925 10:03:06.189915 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.189928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.189980 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.190140 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.190141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.190329 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.190390 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.190392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.191432 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.191433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.192347 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.192351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.192879 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.192880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.192963 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.193499 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.193506 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.195278 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.197656 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.197657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.197658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.197960 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.198195 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.198196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.198197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.198263 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.198572 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.198573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.198624 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.198696 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.198809 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 10:03:06.198935 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.189965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.19.46.252:44641)
0925 10:03:06.190008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.190010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.190019 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:06.190054 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.190055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.190066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.190129 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.190486 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.191323 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.192452 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.192787 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.192866 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.195388 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.195390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.195390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.195399 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.197366 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.198048 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.198049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.198050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.198118 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.198333 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.198334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.198335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.198499 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.198575 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.198744 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.198831 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:03:06.199815 25738 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.46.252:44641: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04"
member_type: NON_VOTER
last_known_addr {
host: "127.19.46.252"
port: 44641
}
attrs {
promote: true
}
}
dest_uuid: "4ecca3724a51473ba1fda458a760c4a1"
cas_config_opid_index: 5
I20260925 10:03:06.200091 25738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } }
I20260925 10:03:06.200446 26305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.200973 26439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:06.201339 26305 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.46.252:44641
I20260925 10:03:06.201406 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.201507 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.202039 26439 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:03:06.202319 26397 master.cc:537] Master@127.19.46.252:44641 shutting down...
I20260925 10:03:06.202337 26304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:03:06.203025 25733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 -> Peer 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252:44641): Couldn't send request to peer 6c195dab1cf947f3b3bfd9141c796c04. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:03:06.207348 26397 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:06.207465 26397 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:06.207523 26397 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: stopping tablet replica
I20260925 10:03:06.207772 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.200642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.19.46.253:42999)
0925 10:03:06.200690 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.200692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.203187 (+ 2495us) server_negotiation.cc:407] Connection header received
0925 10:03:06.203322 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.203323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.203349 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.203394 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.203622 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.204492 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.206189 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.206895 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.206949 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.207018 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.207597 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.207658 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:03:06.208037 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.200534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35096)
0925 10:03:06.203119 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.203132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.203219 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.203481 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.203483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.203485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.203539 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.203541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.204699 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.204700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.206059 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.206062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.207369 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.207370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.207450 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.207477 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.207914 (+ 437us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.207968 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2565}
I20260925 10:03:06.208429 26353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 LEARNER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:03:06.208642 26304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:03:06.209375 26353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.210927 26304 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: attempting to promote NON_VOTER 49895b4e59ff41f9aed9a33380c4b5ae to VOTER
I20260925 10:03:06.210930 26383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecca3724a51473ba1fda458a760c4a1. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.211031 26383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: LEARNER
I20260925 10:03:06.211066 26383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: NON_VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: true } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.211117 26383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: LEARNER
I20260925 10:03:06.211174 26304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } }
I20260925 10:03:06.211916 26353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 LEARNER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:03:06.211951 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.211745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42050)
0925 10:03:06.211858 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.211914 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 10:03:06.212150 26305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:03:06.212489 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.212371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42062)
0925 10:03:06.212439 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.212468 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:03:06.212597 25733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 -> Peer 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252:44641): Couldn't send request to peer 6c195dab1cf947f3b3bfd9141c796c04. Status: Network error: Client connection negotiation failed: client connection to 127.19.46.252:44641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:03:06.213208 26305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.214367 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.214458 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.214715 26397 master.cc:552] Master@127.19.46.252:44641 shutdown complete.
I20260925 10:03:06.214787 26397 master_runner.cc:305] Clearing existing system tablet
I20260925 10:03:06.214753 26304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.214818 26304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.215015 26397 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.215094 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.215127 26353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.216231 26397 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.216506 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.216604 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:03:06.216830 26496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.216941 26397 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.216989 26397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.217011 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "6c195dab1cf947f3b3bfd9141c796c04"
format_stamp: "Formatted at 2026-09-25 10:03:06 on dist-test-slave-f9rl"
server_key: "8df5e7af5744f7cd734b2f6341bb29e0"
server_key_iv: "06ef7ab133321482c922b2cd04986cc0"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.217852 26397 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:06.219287 26397 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:03:06.219355 26397 log.cc:1196] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:03:06.219561 26397 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Deleting consensus metadata
I20260925 10:03:06.219739 26397 master_runner.cc:315] Copying system tablet from 127.19.46.254:44019
I20260925 10:03:06.220332 26397 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: tablet copy: Beginning tablet copy session from remote peer at address 127.19.46.254:44019
I20260925 10:03:06.228502 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.220653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40152)
0925 10:03:06.220785 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.220796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.220846 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.221450 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.221451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.221654 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.221741 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.221743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.222728 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.222729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.223573 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.223576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.224156 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.224157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.224224 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.224753 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.224758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.225171 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.227281 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.227282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.227283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.227526 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.227761 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.227762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.227762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.227842 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.228154 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.228155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.228206 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.228280 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.228393 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260925 10:03:06.228533 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.220732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.19.46.254:44019)
0925 10:03:06.221282 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.221284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.221294 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:06.221323 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.221324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.221335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.221401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.221829 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.222640 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.223726 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.224062 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.224148 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.225266 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.225267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.225268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.225282 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.227180 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.227615 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.227616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.227617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.227685 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.227918 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.227919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.227919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.228076 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.228156 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.228336 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.228424 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
I20260925 10:03:06.229228 25766 tablet_copy_service.cc:140] P 4ecca3724a51473ba1fda458a760c4a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6c195dab1cf947f3b3bfd9141c796c04 ({username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:40152)
I20260925 10:03:06.229295 25766 tablet_copy_service.cc:161] P 4ecca3724a51473ba1fda458a760c4a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6c195dab1cf947f3b3bfd9141c796c04 at {username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:40152: session id = 6c195dab1cf947f3b3bfd9141c796c04-00000000000000000000000000000000
I20260925 10:03:06.230264 25766 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:03:06.230854 26397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:03:06.232324 26397 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: tablet copy: Starting download of 0 data blocks...
I20260925 10:03:06.232491 26397 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: tablet copy: Starting download of 1 WAL segments...
I20260925 10:03:06.233356 26397 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:03:06.234493 25766 tablet_copy_service.cc:342] P 4ecca3724a51473ba1fda458a760c4a1: Request end of tablet copy session 6c195dab1cf947f3b3bfd9141c796c04-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:40152
I20260925 10:03:06.234549 25766 tablet_copy_service.cc:433] P 4ecca3724a51473ba1fda458a760c4a1: ending tablet copy session 6c195dab1cf947f3b3bfd9141c796c04-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6c195dab1cf947f3b3bfd9141c796c04
I20260925 10:03:06.237293 26397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:06.238679 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.242543 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.242760 26397 server_base.cc:1034] running on GCE node
W20260925 10:03:06.245606 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.245947 26397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:06.246168 26397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:06.247315 26397 hybrid_clock.cc:648] HybridClock initialized: now 1790330586247303 us; error 37 us; skew 500 ppm
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:06.248973 26397 init.cc:377] Logged in from keytab as oryx/127.19.46.252@KRBTEST.COM (short username oryx)
I20260925 10:03:06.249254 26397 webserver.cc:460] Webserver started at http://127.19.46.252:32839/ using document root <none> and password file <none>
I20260925 10:03:06.249377 26397 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.249411 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.250217 26397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:06.250835 26513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.251001 26397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:03:06.251055 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "6c195dab1cf947f3b3bfd9141c796c04"
format_stamp: "Formatted at 2026-09-25 10:03:06 on dist-test-slave-f9rl"
server_key: "8df5e7af5744f7cd734b2f6341bb29e0"
server_key_iv: "06ef7ab133321482c922b2cd04986cc0"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.251117 26397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.276155 26397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:06.278528 26397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:06.282234 26397 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.252:44641
I20260925 10:03:06.282672 26397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
I20260925 10:03:06.285660 26565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.252:44641 every 8 connection(s)
I20260925 10:03:06.286566 26566 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:03:06.287168 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Bootstrap starting.
I20260925 10:03:06.290851 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:03:06.291134 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Bootstrap complete.
I20260925 10:03:06.291358 26566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } }
I20260925 10:03:06.291478 26566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c195dab1cf947f3b3bfd9141c796c04, State: Initialized, Role: LEARNER
I20260925 10:03:06.291527 26566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } }
I20260925 10:03:06.291744 26566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:06.291756 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.292013 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:03:06.293886 26580 catalog_manager.cc:1259] Loaded cluster ID: 78aaadd1e3284d56809c19d848002c2c
I20260925 10:03:06.293983 26580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: loading cluster ID for follower catalog manager: success
I20260925 10:03:06.295197 26580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: acquiring CA information for follower catalog manager: success
I20260925 10:03:06.295593 26580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:03:06.300686 26490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:06.300926 26490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:06.300980 26490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:06.302259 26490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:06.302325 26490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:06.302371 26490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:06.303896 26490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:37819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=41567
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:06.304174 26490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:06.304409 26490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:06.309423 26585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.309497 26588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.309492 26586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.310168 26490 server_base.cc:1034] running on GCE node
I20260925 10:03:06.310382 26490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:06.310642 26490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:06.311825 26490 hybrid_clock.cc:648] HybridClock initialized: now 1790330586311824 us; error 39 us; skew 500 ppm
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:06.314419 26490 init.cc:377] Logged in from keytab as oryx/127.19.46.193@KRBTEST.COM (short username oryx)
I20260925 10:03:06.314899 26490 webserver.cc:460] Webserver started at http://127.19.46.193:41567/ using document root <none> and password file <none>
I20260925 10:03:06.315141 26490 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.315258 26490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.316665 26490 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:06.317312 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.317523 26490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:06.317620 26490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "5d0f9049117147d1ab05ffc5627e0df1"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "3ad3c58e05d5872f9deaccc7857697f4"
server_key_iv: "1955c88943d14b29672b9ba29892d481"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.317972 26490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.335884 26490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:06.338385 26490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:06.338586 26490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:06.338848 26490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:06.339193 26490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:06.339250 26490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.339296 26490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:06.339326 26490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.345844 26490 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:37819
I20260925 10:03:06.346019 26708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:37819 every 8 connection(s)
I20260925 10:03:06.346369 26490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:06.354092 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26490
I20260925 10:03:06.354261 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25942
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.359293 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:55035 (local address 127.19.46.252:44641)
0925 10:03:06.347291 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.347294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.347537 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:03:06.348663 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.348665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.348678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.348726 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.349554 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350531 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351562 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.351945 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351987 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.355526 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.355527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.355528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.355547 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.357698 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358149 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.358150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.358150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.358210 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358443 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.358444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.358444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.358593 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.358643 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.359096 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.359188 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260925 10:03:06.359290 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:40553 (local address 127.19.46.253:42999)
0925 10:03:06.347629 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.347631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.347710 (+ 79us) server_negotiation.cc:407] Connection header received
0925 10:03:06.348663 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.348665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.348678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.348712 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.349554 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350322 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351425 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.351713 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351752 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.355614 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.355616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.355616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.355624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.357431 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358054 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.358056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.358056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.358123 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358443 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.358444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.358444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.358600 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.358641 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.359096 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.359188 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:03:06.360076 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:50177 (local address 127.19.46.254:44019)
0925 10:03:06.347314 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.347317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.347698 (+ 381us) server_negotiation.cc:407] Connection header received
0925 10:03:06.348644 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.348645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.348658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.348714 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.349551 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350068 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351178 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.351436 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.356473 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.356475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.356476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.356484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.358783 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.359202 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.359203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.359204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.359263 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.359474 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.359475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.359476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.359626 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.359672 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.359903 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.359987 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:03:06.360188 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:50177)
0925 10:03:06.347624 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.347636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.348538 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.348802 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.348809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.349145 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.349399 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.349403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350189 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.350190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.351081 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.351083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.351541 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.351671 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.352241 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.352252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.356353 (+ 4101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.358886 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.358888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.359122 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.359331 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.359332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.359333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.359393 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.359726 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.359727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.359775 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.359863 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.359984 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":38,"thread_start_us":126,"threads_started":1}
I20260925 10:03:06.360384 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.19.46.193:40553)
0925 10:03:06.347649 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.347659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.348537 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.348801 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.348806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.349173 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.349400 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.349406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350444 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.350445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.351319 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.351322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.351832 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.351832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.351892 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.352370 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.352374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.355529 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.357558 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.357568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.357571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.357963 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.358215 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.358216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.358364 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.358725 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.358730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.358921 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.359655 (+ 734us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.360271 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":516,"thread_start_us":64,"threads_started":1}
I20260925 10:03:06.360664 26710 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:06.360749 26711 heartbeater.cc:347] Connected to a master server at 127.19.46.253:42999
I20260925 10:03:06.360769 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.347159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.19.46.193:55035)
0925 10:03:06.347391 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.347435 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.348535 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.348801 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.348806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.349177 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.349399 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.349406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.350617 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.350618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.351483 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.351485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.352064 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.352065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.352142 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.352573 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.352578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.355382 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.357779 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.357780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.357781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.358065 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.358288 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.358288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.358289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.358364 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.358723 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.358730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.358921 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.359604 (+ 683us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.359845 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":14,"thread_start_us":117,"threads_started":1}
I20260925 10:03:06.360805 26711 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.361100 26711 heartbeater.cc:511] Master 127.19.46.253:42999 requested a full tablet report, sending...
I20260925 10:03:06.361162 26709 heartbeater.cc:347] Connected to a master server at 127.19.46.252:44641
I20260925 10:03:06.361233 26709 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.361300 26709 heartbeater.cc:511] Master 127.19.46.252:44641 requested a full tablet report, sending...
I20260925 10:03:06.360769 26710 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.361572 26710 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:06.361634 26343 ts_manager.cc:194] Registered new tserver with Master: 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819)
I20260925 10:03:06.361640 26530 ts_manager.cc:194] Registered new tserver with Master: 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819)
I20260925 10:03:06.361820 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:33811
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:44019
--webserver_port=42157
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:03:06.361832 25738 ts_manager.cc:194] Re-registered known tserver with Master: 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819)
I20260925 10:03:06.362335 25738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:50177
W20260925 10:03:06.449424 26717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:06.449704 26717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:06.449731 26717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:06.450871 26717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:06.450917 26717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:06.450973 26717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:06.452528 26717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:33811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=42157
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:06.452735 26717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:06.452924 26717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:06.458238 26723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.458353 26725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.458362 26722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.458431 26717 server_base.cc:1034] running on GCE node
I20260925 10:03:06.458673 26717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:06.458884 26717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:06.460052 26717 hybrid_clock.cc:648] HybridClock initialized: now 1790330586460009 us; error 57 us; skew 500 ppm
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:06.462883 26717 init.cc:377] Logged in from keytab as oryx/127.19.46.194@KRBTEST.COM (short username oryx)
I20260925 10:03:06.463382 26717 webserver.cc:460] Webserver started at http://127.19.46.194:42157/ using document root <none> and password file <none>
I20260925 10:03:06.463586 26717 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.463637 26717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.464954 26717 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:06.465618 26732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.465870 26717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:03:06.465973 26717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "813640b15f584c4e94b1fdc9d52ee31c"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "9fcfbca3c81193655a185704a8c62c5e"
server_key_iv: "2410a0f1d7c7c25f142131fa1f6a58bf"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.466328 26717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.478591 26717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:06.481304 26717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:06.481490 26717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:06.481715 26717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:06.482067 26717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:06.482105 26717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.482128 26717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:06.482141 26717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.488240 26717 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:33811
I20260925 10:03:06.488296 26845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:33811 every 8 connection(s)
I20260925 10:03:06.488739 26717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.497664 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26717
I20260925 10:03:06.497857 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26076
I20260925 10:03:06.502990 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:53095 (local address 127.19.46.252:44641)
0925 10:03:06.489965 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.489967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.489976 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:06.490831 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.490832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.490844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.490894 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.491802 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.492686 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494437 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494753 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494866 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.499248 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.499249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.499250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.499260 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.501099 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502048 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.502050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.502050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.502117 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502363 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.502365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.502365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.502512 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.502573 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.502787 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.502885 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:03:06.503088 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52567 (local address 127.19.46.253:42999)
0925 10:03:06.489956 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.489959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.489969 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:06.490981 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.490982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.490994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.491027 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.491801 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.492674 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494239 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494523 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494736 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.497157 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.497159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.497159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.497167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.500328 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502104 (+ 1776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.502106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.502106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.502195 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502428 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.502430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.502430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.502588 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.502634 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.502839 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.502948 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:03:06.503302 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.19.46.194:53095)
0925 10:03:06.489595 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.489638 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.490701 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.490974 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.490978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.491432 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.491646 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.491652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.492765 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.492765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.494351 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.494352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494924 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.495001 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.495453 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.495457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.497927 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.501215 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.501217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.501217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.501930 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.502199 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.502200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.502272 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.502608 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.502609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.502648 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.503088 (+ 440us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.503193 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":21,"thread_start_us":108,"threads_started":1}
I20260925 10:03:06.503465 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39137 (local address 127.19.46.254:44019)
0925 10:03:06.489516 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.489519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.489721 (+ 202us) server_negotiation.cc:407] Connection header received
0925 10:03:06.490963 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.490964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.490978 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.491036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.491756 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.492323 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.493831 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494040 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494111 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.496796 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.496798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.496798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.496807 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.499999 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.500794 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.500796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.500796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.500868 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.501327 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.501328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.501329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.501554 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.501652 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.503265 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.503374 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:03:06.503504 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.19.46.194:52567)
0925 10:03:06.489863 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.489873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.490858 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.491096 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.491097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.491422 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.491646 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.491650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.493492 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.493493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.494148 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.494149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494792 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.494863 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.495292 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.495297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.497013 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.501208 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.501209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.501210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.501483 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.502274 (+ 791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.502275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.502275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.502346 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.502681 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.502682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.502726 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.503221 (+ 495us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.503349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260925 10:03:06.503877 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.489427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:39137)
0925 10:03:06.489596 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.489642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.490858 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.491060 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.491060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.491420 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.491646 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.491652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.492422 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.492423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.493727 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.493729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.494134 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.494135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.494283 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.494868 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.494877 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.496579 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.500114 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.500128 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.500395 (+ 267us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.500683 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.500946 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.500948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.500948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.501224 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.502299 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.502310 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.502523 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.502650 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.502876 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":11,"thread_start_us":80,"threads_started":1}
I20260925 10:03:06.504132 26848 heartbeater.cc:347] Connected to a master server at 127.19.46.253:42999
I20260925 10:03:06.504189 26846 heartbeater.cc:347] Connected to a master server at 127.19.46.252:44641
I20260925 10:03:06.504254 26846 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.504256 26848 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.504308 26847 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:06.504343 26847 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.504472 26848 heartbeater.cc:511] Master 127.19.46.253:42999 requested a full tablet report, sending...
I20260925 10:03:06.504472 26846 heartbeater.cc:511] Master 127.19.46.252:44641 requested a full tablet report, sending...
I20260925 10:03:06.504470 26847 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:06.504911 25738 ts_manager.cc:194] Re-registered known tserver with Master: 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811)
I20260925 10:03:06.504974 26530 ts_manager.cc:194] Registered new tserver with Master: 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811)
I20260925 10:03:06.505023 26343 ts_manager.cc:194] Registered new tserver with Master: 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811)
I20260925 10:03:06.505482 25738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:39137
I20260925 10:03:06.505838 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:40869
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:44019
--webserver_port=43833
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:06.589879 26853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:06.590163 26853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:06.590209 26853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:06.591405 26853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:06.591483 26853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:06.591529 26853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:06.593161 26853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:40869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=43833
--tserver_master_addrs=127.19.46.254:44019,127.19.46.253:42999,127.19.46.252:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:06.593470 26853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:06.593726 26853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:03:06.599826 26853 server_base.cc:1034] running on GCE node
W20260925 10:03:06.599800 26858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.599834 26859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:06.599805 26861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:06.600447 26853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:06.600723 26853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:06.601899 26853 hybrid_clock.cc:648] HybridClock initialized: now 1790330586601879 us; error 45 us; skew 500 ppm
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:06.604488 26853 init.cc:377] Logged in from keytab as oryx/127.19.46.195@KRBTEST.COM (short username oryx)
I20260925 10:03:06.604969 26853 webserver.cc:460] Webserver started at http://127.19.46.195:43833/ using document root <none> and password file <none>
I20260925 10:03:06.605162 26853 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:06.605214 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:06.606614 26853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:06.607295 26868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.607575 26853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:06.607671 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "fbd72b40b5ed4a15975b5e46061b0747"
format_stamp: "Formatted at 2026-09-25 10:03:05 on dist-test-slave-f9rl"
server_key: "ed73db7369e1549c5e76a8bfc530c3a9"
server_key_iv: "65ca7b3ef8ca3fc4cffc595f4790488f"
server_key_version: "encryptionkey@0"
I20260925 10:03:06.608114 26853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:06.639074 26853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:06.641332 26853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:06.641549 26853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:06.641803 26853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:06.642140 26853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:06.642174 26853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.642197 26853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:06.642230 26853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:06.648917 26853 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:40869
I20260925 10:03:06.649012 26981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:40869 every 8 connection(s)
I20260925 10:03:06.649355 26853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:06.651412 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 26853
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.253@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
Sep 25 10:03:06 dist-test-slave-f9rl krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330586, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.252@KRBTEST.COM
I20260925 10:03:06.661048 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.652606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.19.46.254:44019)
0925 10:03:06.653246 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.653248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.653367 (+ 119us) server_negotiation.cc:407] Connection header received
0925 10:03:06.653425 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.653426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.653462 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.653533 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.654338 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.655042 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.656031 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.656307 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.656363 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.657914 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.657916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.657926 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.657933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.659421 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.659839 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.659840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.659841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.659902 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.660138 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.660139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.660139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.660285 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.660384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.660813 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.660901 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":81,"threads_started":1}
I20260925 10:03:06.663944 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:36195 (local address 127.19.46.252:44641)
0925 10:03:06.650216 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.650219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.651353 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 10:03:06.651462 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.651463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.651474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.651535 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.652549 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.653313 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.656905 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.657254 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.657328 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.659863 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.659865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.659865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.659873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.661713 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.662329 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.662330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.662331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.662392 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.662665 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.662666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.662666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.662808 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.662874 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.663741 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.663845 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:03:06.664312 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.19.46.195:36195)
0925 10:03:06.651285 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.651297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.651383 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.651587 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.651587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.652116 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.652410 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.652443 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.653497 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.653498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.655972 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.655974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.657393 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.657394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.657497 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.657915 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.657920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.659740 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.661834 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.661843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.661846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.662241 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.662460 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.662461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.662461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.662597 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.662905 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.662911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.663085 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.663201 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.663389 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":1423,"spinlock_wait_cycles":7040,"thread_start_us":166,"threads_started":1}
I20260925 10:03:06.663086 25738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:03:06.665274 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:57641 (local address 127.19.46.253:42999)
0925 10:03:06.650700 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.650702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.651702 (+ 1000us) server_negotiation.cc:407] Connection header received
0925 10:03:06.651820 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.651821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.651834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.651866 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.652533 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.653367 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.655229 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.655565 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.655631 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.659976 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.659978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.659979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.659988 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.661854 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.664307 (+ 2453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.664308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.664309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.664378 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.664733 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.664734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.664734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.664880 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.664969 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.665108 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.665195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:03:06.665503 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.19.46.195:57641)
0925 10:03:06.651632 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.651643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.651733 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.651935 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.651936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.652250 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.652410 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.652418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.653505 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.653506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.655139 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.655142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.655698 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.655699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.655847 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.656448 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.656459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.659850 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.662088 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.662089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.662089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.664225 (+ 2136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.664581 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.664582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.664582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.664656 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.664960 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.664961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.665005 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.665082 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.665184 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":75,"thread_start_us":77,"threads_started":1}
W20260925 10:03:06.665482 25738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:06.666033 26982 heartbeater.cc:347] Connected to a master server at 127.19.46.252:44641
I20260925 10:03:06.666141 26982 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.666383 26984 heartbeater.cc:347] Connected to a master server at 127.19.46.253:42999
I20260925 10:03:06.666441 26982 heartbeater.cc:511] Master 127.19.46.252:44641 requested a full tablet report, sending...
I20260925 10:03:06.666446 26984 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.666561 26984 heartbeater.cc:511] Master 127.19.46.253:42999 requested a full tablet report, sending...
I20260925 10:03:06.666920 26343 ts_manager.cc:194] Registered new tserver with Master: fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:06.666958 26530 ts_manager.cc:194] Registered new tserver with Master: fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:06.668555 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:39343 (local address 127.19.46.254:44019)
0925 10:03:06.650297 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.650299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.650624 (+ 325us) server_negotiation.cc:407] Connection header received
0925 10:03:06.651191 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.651192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.651203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.651305 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.654009 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.654782 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.655793 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.656084 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.656119 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.659342 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.659344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.659344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.659351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.661755 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.667598 (+ 5843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.667600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.667600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.667663 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.667879 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.667880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.667880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.668074 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.668161 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.668364 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.668460 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:03:06.669205 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.650167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:39343)
0925 10:03:06.650455 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.650493 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.651074 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.651454 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.651459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.652115 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:06.653901 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.653903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.654871 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.654872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.655702 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.655705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.656190 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.656191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.656273 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.656685 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:06.656690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:06.659233 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:06.661855 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.661856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.661857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.667492 (+ 5635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.667743 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:06.667744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:06.667744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:06.667807 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:06.668196 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:06.668197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:06.668240 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.668679 (+ 439us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.668791 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":2,"thread_start_us":173,"threads_started":1}
I20260925 10:03:06.669605 26983 heartbeater.cc:347] Connected to a master server at 127.19.46.254:44019
I20260925 10:03:06.669659 26983 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:06.669737 26983 heartbeater.cc:511] Master 127.19.46.254:44019 requested a full tablet report, sending...
I20260925 10:03:06.670332 25746 ts_manager.cc:194] Re-registered known tserver with Master: fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:06.670818 25746 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:39343
I20260925 10:03:06.671003 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.653174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.19.46.252:44641)
0925 10:03:06.653690 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.653692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.654095 (+ 403us) server_negotiation.cc:407] Connection header received
0925 10:03:06.654118 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.654119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.654132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.654188 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.662378 (+ 8190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.663096 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.664184 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.664566 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.664628 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.667046 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.667048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.667048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.667056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.669104 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.669886 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.669888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.669888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.669961 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.670159 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.670160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.670160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.670294 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.670698 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.670745 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.670820 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":51,"threads_started":1}
I20260925 10:03:06.672160 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.652939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.19.46.253:42999)
0925 10:03:06.653989 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.653991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.653999 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:06.654019 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.654019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.654031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.656090 (+ 2059us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.656130 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.662755 (+ 6625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.663901 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.665531 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.665884 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.667132 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.667133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.667134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.667142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.669493 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.670302 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.670304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.670304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.670373 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.670617 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.670618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.670618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.670864 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.671930 (+ 1066us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.671972 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.672062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":61,"threads_started":1}
I20260925 10:03:06.676324 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:33811 (local address 127.0.0.1:43332)
0925 10:03:06.672345 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.672357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.672398 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.672991 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.672992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.672994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.673042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.673044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674211 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.674213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.675129 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.675131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.676070 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.676071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.676134 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.676138 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.676187 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.676231 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 10:03:06.676622 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.19.46.194:33811)
0925 10:03:06.672802 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.672807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.672818 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:06.672859 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.672861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.672877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.673121 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.673187 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674114 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.675227 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.675960 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.676034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.676450 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.676496 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.676541 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260925 10:03:06.677414 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:37819 (local address 127.0.0.1:43554)
0925 10:03:06.673153 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.673163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.673220 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.673428 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.673429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.673477 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.673517 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.673519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674942 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.674943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.676381 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.676385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.677153 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.677154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.677215 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.677220 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.677280 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.677331 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":30,"thread_start_us":93,"threads_started":1}
I20260925 10:03:06.677567 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:40869 (local address 127.0.0.1:60818)
0925 10:03:06.673112 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.673125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.673181 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.673550 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.673551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.673554 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.673605 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.673606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674991 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.674992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.675819 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.675821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.677295 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.677296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.677364 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.677369 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.677424 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.677492 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":113,"threads_started":1}
I20260925 10:03:06.677610 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.19.46.193:37819)
0925 10:03:06.672732 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.672737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.673239 (+ 502us) server_negotiation.cc:407] Connection header received
0925 10:03:06.673303 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.673306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.673324 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.673420 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.673591 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674851 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.676480 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.677056 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.677151 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.677223 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.677294 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.677331 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260925 10:03:06.677492 26780 tablet_service.cc:1467] Processing CreateTablet for tablet be360844939f40949f89dfdfe7857324 (DEFAULT_TABLE table=test-table [id=e41174d668a8455089e4891b84a3fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:06.677812 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.672806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.19.46.195:40869)
0925 10:03:06.673328 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.673332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.673341 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:06.673378 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.673380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.673397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.673491 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.673690 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.674925 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.675936 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.677105 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.677178 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.677255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.677669 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.677718 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":127,"threads_started":1}
I20260925 10:03:06.677901 26780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be360844939f40949f89dfdfe7857324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:06.678151 26643 tablet_service.cc:1467] Processing CreateTablet for tablet be360844939f40949f89dfdfe7857324 (DEFAULT_TABLE table=test-table [id=e41174d668a8455089e4891b84a3fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:06.678442 26643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be360844939f40949f89dfdfe7857324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:06.678582 26916 tablet_service.cc:1467] Processing CreateTablet for tablet be360844939f40949f89dfdfe7857324 (DEFAULT_TABLE table=test-table [id=e41174d668a8455089e4891b84a3fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:06.678910 26916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be360844939f40949f89dfdfe7857324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:06.681748 27009 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Bootstrap starting.
I20260925 10:03:06.681829 27010 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Bootstrap starting.
I20260925 10:03:06.682503 27009 tablet_bootstrap.cc:654] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:06.682614 27010 tablet_bootstrap.cc:654] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:06.682756 27011 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Bootstrap starting.
I20260925 10:03:06.682847 27009 log.cc:824] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:06.682932 27010 log.cc:824] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:06.683439 27011 tablet_bootstrap.cc:654] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:06.683660 27009 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: No bootstrap required, opened a new log
I20260925 10:03:06.683741 27009 ts_tablet_manager.cc:1397] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:03:06.683753 27011 log.cc:824] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:06.683781 27010 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: No bootstrap required, opened a new log
I20260925 10:03:06.683909 27010 ts_tablet_manager.cc:1397] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:03:06.684712 27011 tablet_bootstrap.cc:492] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: No bootstrap required, opened a new log
I20260925 10:03:06.684789 27011 ts_tablet_manager.cc:1397] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:03:06.685941 27009 raft_consensus.cc:348] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.686090 27009 raft_consensus.cc:374] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:06.686143 27009 raft_consensus.cc:729] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0f9049117147d1ab05ffc5627e0df1, State: Initialized, Role: FOLLOWER
I20260925 10:03:06.686285 27009 consensus_queue.cc:260] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.686595 27009 ts_tablet_manager.cc:1428] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:06.686537 27010 raft_consensus.cc:348] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.686718 26710 heartbeater.cc:503] Master 127.19.46.254:44019 was elected leader, sending a full tablet report...
I20260925 10:03:06.686741 27010 raft_consensus.cc:374] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:06.686817 27010 raft_consensus.cc:729] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813640b15f584c4e94b1fdc9d52ee31c, State: Initialized, Role: FOLLOWER
I20260925 10:03:06.686795 27011 raft_consensus.cc:348] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.686935 27011 raft_consensus.cc:374] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:06.686985 27011 raft_consensus.cc:729] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd72b40b5ed4a15975b5e46061b0747, State: Initialized, Role: FOLLOWER
I20260925 10:03:06.687006 27010 consensus_queue.cc:260] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.687109 27011 consensus_queue.cc:260] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.687295 27010 ts_tablet_manager.cc:1428] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:06.687345 27011 ts_tablet_manager.cc:1428] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:06.687716 26983 heartbeater.cc:503] Master 127.19.46.254:44019 was elected leader, sending a full tablet report...
I20260925 10:03:06.687999 26847 heartbeater.cc:503] Master 127.19.46.254:44019 was elected leader, sending a full tablet report...
I20260925 10:03:06.699194 27015 raft_consensus.cc:482] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:06.699383 27015 raft_consensus.cc:504] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.700104 27015 leader_election.cc:288] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811), fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:06.703281 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.699524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.19.46.252:44641)
0925 10:03:06.699580 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.699583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.699594 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:06.699650 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.699651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.699681 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.699754 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.699908 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.700652 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.702462 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.702920 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.702996 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703071 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.703157 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.703203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:03:06.703310 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.699463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42078)
0925 10:03:06.699523 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.699534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.699585 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.699767 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.699768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.699770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.699837 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.699839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.700797 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.700798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.702351 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.702355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.703009 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.703010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.703080 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703084 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.703146 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.703209 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:03:06.703550 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.700217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:33811 (local address 127.19.46.193:41991)
0925 10:03:06.700282 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.700295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.700341 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.700554 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.700555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.700557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.700651 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.700653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.701613 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.701614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.702380 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.702383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.703260 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.703261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.703349 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703354 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.703413 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.703465 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:03:06.703680 26540 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 10:03:06.703850 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.700243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:40869 (local address 127.19.46.193:38365)
0925 10:03:06.700314 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:06.700324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:06.700359 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:06.700650 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:06.700650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:06.700652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:06.700699 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.700700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.701600 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.701601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.702829 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:06.702832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.703544 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.703545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:06.703632 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703636 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:06.703698 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:03:06.703748 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":27}
I20260925 10:03:06.704121 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.700465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38365 (local address 127.19.46.195:40869)
0925 10:03:06.700534 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.700536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.700544 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:06.700567 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.700569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.700577 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.700628 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.700756 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.701504 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.702930 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.703471 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.703548 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703557 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.703998 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.704051 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:03:06.704478 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.700378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:41991 (local address 127.19.46.194:33811)
0925 10:03:06.700442 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.700444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.700453 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:06.700480 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.700480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.700490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.700537 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:06.700717 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.701524 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.702488 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.703147 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.703214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.703223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.704344 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.704396 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:03:06.704598 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecca3724a51473ba1fda458a760c4a1. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.704607 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be360844939f40949f89dfdfe7857324" candidate_uuid: "5d0f9049117147d1ab05ffc5627e0df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd72b40b5ed4a15975b5e46061b0747" is_pre_election: true
I20260925 10:03:06.704690 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:03:06.704767 26936 raft_consensus.cc:2454] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d0f9049117147d1ab05ffc5627e0df1 in term 0.
I20260925 10:03:06.704981 26800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be360844939f40949f89dfdfe7857324" candidate_uuid: "5d0f9049117147d1ab05ffc5627e0df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813640b15f584c4e94b1fdc9d52ee31c" is_pre_election: true
I20260925 10:03:06.705055 26599 leader_election.cc:302] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d0f9049117147d1ab05ffc5627e0df1, fbd72b40b5ed4a15975b5e46061b0747; no voters:
I20260925 10:03:06.705128 26800 raft_consensus.cc:2454] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d0f9049117147d1ab05ffc5627e0df1 in term 0.
I20260925 10:03:06.705188 27015 raft_consensus.cc:2781] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:06.705259 27015 raft_consensus.cc:482] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:06.705291 27015 raft_consensus.cc:3037] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:06.706055 27015 raft_consensus.cc:504] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.706264 27015 leader_election.cc:288] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [CANDIDATE]: Term 1 election: Requested vote from peers 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811), fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:06.706369 26800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be360844939f40949f89dfdfe7857324" candidate_uuid: "5d0f9049117147d1ab05ffc5627e0df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813640b15f584c4e94b1fdc9d52ee31c"
I20260925 10:03:06.706475 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be360844939f40949f89dfdfe7857324" candidate_uuid: "5d0f9049117147d1ab05ffc5627e0df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd72b40b5ed4a15975b5e46061b0747"
I20260925 10:03:06.706467 26800 raft_consensus.cc:3037] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:06.706561 26936 raft_consensus.cc:3037] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:06.707240 26936 raft_consensus.cc:2454] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d0f9049117147d1ab05ffc5627e0df1 in term 1.
I20260925 10:03:06.707417 26800 raft_consensus.cc:2454] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d0f9049117147d1ab05ffc5627e0df1 in term 1.
I20260925 10:03:06.707455 26599 leader_election.cc:302] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d0f9049117147d1ab05ffc5627e0df1, fbd72b40b5ed4a15975b5e46061b0747; no voters:
I20260925 10:03:06.707589 27015 raft_consensus.cc:2781] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:06.707755 27015 raft_consensus.cc:686] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 LEADER]: Becoming Leader. State: Replica: 5d0f9049117147d1ab05ffc5627e0df1, State: Running, Role: LEADER
I20260925 10:03:06.707881 27015 consensus_queue.cc:237] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } }
I20260925 10:03:06.708730 25746 catalog_manager.cc:5495] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "5d0f9049117147d1ab05ffc5627e0df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } health_report { overall_health: HEALTHY } } }
I20260925 10:03:06.709378 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: NON_VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: true } } }
I20260925 10:03:06.709517 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.710351 26305 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: attempting to promote NON_VOTER 6c195dab1cf947f3b3bfd9141c796c04 to VOTER
I20260925 10:03:06.710481 26305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } }
I20260925 10:03:06.711021 26540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 LEARNER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 10:03:06.711062 26353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Refusing update from remote peer 4ecca3724a51473ba1fda458a760c4a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 10:03:06.711282 26494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 10:03:06.711362 26304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 10:03:06.712023 26305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.712931 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.712951 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.713009 26494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.713114 26494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.713022 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.713032 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:06.713174 26353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.713198 26540 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.714108 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.714107 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ecca3724a51473ba1fda458a760c4a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:06.714197 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:03:06.714198 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:03:06.724068 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.721720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.19.46.193:37819)
0925 10:03:06.721765 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.721767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.721775 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:06.721812 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.721813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.721838 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.721915 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:06.722085 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.722581 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.723309 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.723512 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.723589 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.723726 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:06.723831 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:06.723926 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.723945 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.723972 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:03:06.726040 26800 raft_consensus.cc:1261] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Refusing update from remote peer 5d0f9049117147d1ab05ffc5627e0df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:06.726369 26936 raft_consensus.cc:1261] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Refusing update from remote peer 5d0f9049117147d1ab05ffc5627e0df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:06.726431 27015 consensus_queue.cc:1035] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:06.726653 27022 consensus_queue.cc:1035] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:06.729571 27025 mvcc.cc:205] Tried to move back new op lower bound from 7333194083226783744 to 7333194083156127744. Current Snapshot: MvccSnapshot[applied={T|T < 7333194083226783744}]
I20260925 10:03:06.730611 27026 mvcc.cc:205] Tried to move back new op lower bound from 7333194083226783744 to 7333194083156127744. Current Snapshot: MvccSnapshot[applied={T|T < 7333194083226783744}]
I20260925 10:03:06.732578 25738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:06.732681 25738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:06.734153 25738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ecca3724a51473ba1fda458a760c4a1: Sending DeleteTablet for 3 replicas of tablet be360844939f40949f89dfdfe7857324
I20260925 10:03:06.734527 26780 tablet_service.cc:1514] Processing DeleteTablet for tablet be360844939f40949f89dfdfe7857324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:06 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:43332
I20260925 10:03:06.734699 26916 tablet_service.cc:1514] Processing DeleteTablet for tablet be360844939f40949f89dfdfe7857324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:06 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:60818
I20260925 10:03:06.734835 26643 tablet_service.cc:1514] Processing DeleteTablet for tablet be360844939f40949f89dfdfe7857324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:06 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:43554
I20260925 10:03:06.734917 27035 tablet_replica.cc:330] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: stopping tablet replica
I20260925 10:03:06.735014 27035 raft_consensus.cc:2229] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:06.735190 27035 raft_consensus.cc:2258] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:06.735703 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 25714
I20260925 10:03:06.736990 27035 ts_tablet_manager.cc:1905] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:06.739010 27035 ts_tablet_manager.cc:1918] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:06.739094 27035 log.cc:1196] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/be360844939f40949f89dfdfe7857324
I20260925 10:03:06.739372 27035 ts_tablet_manager.cc:1939] T be360844939f40949f89dfdfe7857324 P fbd72b40b5ed4a15975b5e46061b0747: Deleting consensus metadata
W20260925 10:03:06.742190 26869 connection.cc:535] server connection from 127.0.0.1:60818 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:03:06.742796 27034 tablet_replica.cc:330] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: stopping tablet replica
I20260925 10:03:06.742930 27034 raft_consensus.cc:2229] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:06.743074 27034 raft_consensus.cc:2258] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:06.743266 27036 tablet_replica.cc:330] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: stopping tablet replica
I20260925 10:03:06.743374 27036 raft_consensus.cc:2229] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:06.743530 27036 raft_consensus.cc:2258] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:06.744207 27034 ts_tablet_manager.cc:1905] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:06.744714 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.735186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.19.46.252:44641)
0925 10:03:06.735230 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.735232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.735392 (+ 160us) server_negotiation.cc:407] Connection header received
0925 10:03:06.735496 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.735497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.735507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.735537 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.736614 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.737393 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.738585 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.738888 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.738951 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.740842 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.740843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.740844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.740852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.742730 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.743300 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.743302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.743302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.743364 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.743788 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.743789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.743790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.743949 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.744190 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.744528 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.744615 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:03:06.746114 27034 ts_tablet_manager.cc:1918] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:06.749768 27036 ts_tablet_manager.cc:1905] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:06.750633 27034 log.cc:1196] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/be360844939f40949f89dfdfe7857324
I20260925 10:03:06.751011 27034 ts_tablet_manager.cc:1939] T be360844939f40949f89dfdfe7857324 P 813640b15f584c4e94b1fdc9d52ee31c: Deleting consensus metadata
W20260925 10:03:06.751410 26736 connection.cc:441] server connection from 127.0.0.1:43332 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43332 (request call id 1) could send its response
I20260925 10:03:06.752779 27036 ts_tablet_manager.cc:1918] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:06.752859 27036 log.cc:1196] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/be360844939f40949f89dfdfe7857324
I20260925 10:03:06.752918 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.744346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.19.46.252:44641)
0925 10:03:06.744384 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.744385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.744625 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:03:06.744718 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.744719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.744729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.744760 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.745394 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.746246 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.747365 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.747680 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.747758 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.749418 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.749419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.749420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.749427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.751449 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.752044 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.752045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.752046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.752102 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.752297 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.752298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.752298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.752420 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.752495 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.752732 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.752809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:03:06.753150 27036 ts_tablet_manager.cc:1939] T be360844939f40949f89dfdfe7857324 P 5d0f9049117147d1ab05ffc5627e0df1: Deleting consensus metadata
I20260925 10:03:06.753185 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:06.744137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.19.46.253:42999)
0925 10:03:06.744181 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:06.744184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:06.744279 (+ 95us) server_negotiation.cc:407] Connection header received
0925 10:03:06.744383 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:06.744384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:06.744401 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:06.744432 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:06.745571 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.746321 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.747406 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:06.747713 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:06.747775 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:06.749471 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:06.749473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:06.749474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:06.749483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:06.751592 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.752110 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.752112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.752112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.752169 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:06.752370 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:06.752370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:06.752371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:06.752532 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:06.752611 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:06.752964 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 10:03:06.753056 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:03:06.753530 26598 connection.cc:441] server connection from 127.0.0.1:43554 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43554 (request call id 1) could send its response
I20260925 10:03:07.731415 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:07.731099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:51665)
0925 10:03:07.731268 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:07.731362 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260925 10:03:07.731680 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:07.731422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:55763)
0925 10:03:07.731545 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:07.731634 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
W20260925 10:03:07.731776 26983 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:44019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:07.732034 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:07.731661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:47709)
0925 10:03:07.731922 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:07.731991 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":81,"threads_started":1}
W20260925 10:03:07.732209 26847 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:44019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
W20260925 10:03:07.732450 26710 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:44019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.235018 27047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecca3724a51473ba1fda458a760c4a1)
I20260925 10:03:08.235188 27047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } }
I20260925 10:03:08.235493 27048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecca3724a51473ba1fda458a760c4a1)
I20260925 10:03:08.235543 27047 leader_election.cc:288] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019), 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253:42999)
I20260925 10:03:08.235672 27048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } }
I20260925 10:03:08.235869 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.235666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40646)
0925 10:03:08.235777 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.235825 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":73,"threads_started":1}
W20260925 10:03:08.236102 26514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.236263 27048 leader_election.cc:288] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019), 6c195dab1cf947f3b3bfd9141c796c04 (127.19.46.252:44641)
I20260925 10:03:08.236289 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40650)
0925 10:03:08.236193 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.236234 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
W20260925 10:03:08.236625 26327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.237071 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40654)
0925 10:03:08.237010 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.237040 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:03:08.237154 26327 leader_election.cc:334] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019): Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.239192 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40652)
0925 10:03:08.239095 (+ 2397us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.239148 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2375,"thread_start_us":76,"threads_started":1}
W20260925 10:03:08.239354 26514 leader_election.cc:334] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019): Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.239910 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.235959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42999 (local address 127.0.0.1:35108)
0925 10:03:08.236081 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.236091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.236154 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.236461 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.236462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.236465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.236523 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.236525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.237525 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.237526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.238835 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.238838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.239592 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.239593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.239678 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.239685 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.239748 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.239814 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":3}
I20260925 10:03:08.240150 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.19.46.253:42999)
0925 10:03:08.236302 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.236304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.236312 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:08.236339 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.236340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.236352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.236442 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.236592 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.237419 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.238935 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.239516 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.239588 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.239599 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.240015 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.240068 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":44,"threads_started":1}
I20260925 10:03:08.240348 26353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c195dab1cf947f3b3bfd9141c796c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" is_pre_election: true
I20260925 10:03:08.240473 26353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c195dab1cf947f3b3bfd9141c796c04 in term 1.
I20260925 10:03:08.240648 26517 leader_election.cc:302] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49895b4e59ff41f9aed9a33380c4b5ae, 6c195dab1cf947f3b3bfd9141c796c04; no voters: 4ecca3724a51473ba1fda458a760c4a1
I20260925 10:03:08.240758 27047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:03:08.240834 27047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4ecca3724a51473ba1fda458a760c4a1)
I20260925 10:03:08.240865 27047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:03:08.241900 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.19.46.252:44641)
0925 10:03:08.237495 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.237497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.237508 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:08.237533 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.237535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.237547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.237797 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.237841 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.238664 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.240907 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.241542 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.241614 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.241626 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.241785 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.241825 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":44,"threads_started":1}
I20260925 10:03:08.241900 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.236373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:44641 (local address 127.0.0.1:42104)
0925 10:03:08.236452 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.236463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.236535 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.237662 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.237664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.237666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.237715 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.237717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.238771 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.238773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.240813 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.240816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.241616 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.241617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.241679 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.241683 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.241778 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.241824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":47,"thread_start_us":44,"threads_started":1}
I20260925 10:03:08.241997 27047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } }
I20260925 10:03:08.242175 27047 leader_election.cc:288] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 election: Requested vote from peers 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019), 49895b4e59ff41f9aed9a33380c4b5ae (127.19.46.253:42999)
I20260925 10:03:08.242363 26353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c195dab1cf947f3b3bfd9141c796c04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49895b4e59ff41f9aed9a33380c4b5ae"
I20260925 10:03:08.242450 26353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:03:08.242457 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.242250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40660)
0925 10:03:08.242374 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.242407 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 10:03:08.242471 26540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6c195dab1cf947f3b3bfd9141c796c04" is_pre_election: true
I20260925 10:03:08.242635 26540 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49895b4e59ff41f9aed9a33380c4b5ae in current term 2: Already voted for candidate 6c195dab1cf947f3b3bfd9141c796c04 in this term.
I20260925 10:03:08.243309 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.243213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40672)
0925 10:03:08.243259 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.243285 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:03:08.243314 26330 leader_election.cc:302] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49895b4e59ff41f9aed9a33380c4b5ae; no voters: 4ecca3724a51473ba1fda458a760c4a1, 6c195dab1cf947f3b3bfd9141c796c04
I20260925 10:03:08.243347 26353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c195dab1cf947f3b3bfd9141c796c04 in term 2.
I20260925 10:03:08.243456 27048 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:03:08.243525 26517 leader_election.cc:302] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49895b4e59ff41f9aed9a33380c4b5ae, 6c195dab1cf947f3b3bfd9141c796c04; no voters:
I20260925 10:03:08.243597 27047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:03:08.243767 27047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [term 2 LEADER]: Becoming Leader. State: Replica: 6c195dab1cf947f3b3bfd9141c796c04, State: Running, Role: LEADER
W20260925 10:03:08.243825 26514 leader_election.cc:334] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019): Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
I20260925 10:03:08.243868 27047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } }
I20260925 10:03:08.244254 27057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c195dab1cf947f3b3bfd9141c796c04. Latest consensus state: current_term: 2 leader_uuid: "6c195dab1cf947f3b3bfd9141c796c04" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:08.244410 27057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:08.244642 27059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:08.245164 27059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:08.245363 27059 catalog_manager.cc:1259] Loaded cluster ID: 78aaadd1e3284d56809c19d848002c2c
I20260925 10:03:08.245424 27059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:08.245779 27059 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:08.245883 27059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Loaded TSK: 0
I20260925 10:03:08.246131 27059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:08.298645 26530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:08.300396 26530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:08.301580 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.301416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40690)
0925 10:03:08.301513 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.301544 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 10:03:08.301586 26353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [term 2 FOLLOWER]: Refusing update from remote peer 6c195dab1cf947f3b3bfd9141c796c04: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 10:03:08.301947 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.301815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.0.0.1:40702)
0925 10:03:08.301904 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.301928 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:03:08.301899 27057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 10:03:08.302064 26514 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 -> Peer 4ecca3724a51473ba1fda458a760c4a1 (127.19.46.254:44019): Couldn't send request to peer 4ecca3724a51473ba1fda458a760c4a1. Status: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:03:08.302587 27048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c195dab1cf947f3b3bfd9141c796c04. Latest consensus state: current_term: 2 leader_uuid: "6c195dab1cf947f3b3bfd9141c796c04" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:08.302695 27048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:03:08.303150 27057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6c195dab1cf947f3b3bfd9141c796c04" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:08.303253 27057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:08.303233 27048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6c195dab1cf947f3b3bfd9141c796c04" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ecca3724a51473ba1fda458a760c4a1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 44019 } } peers { permanent_uuid: "49895b4e59ff41f9aed9a33380c4b5ae" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42999 } attrs { promote: false } } peers { permanent_uuid: "6c195dab1cf947f3b3bfd9141c796c04" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 44641 } attrs { promote: false } } }
I20260925 10:03:08.303320 27048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49895b4e59ff41f9aed9a33380c4b5ae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:03:08.308466 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:33811 (local address 127.0.0.1:43340)
0925 10:03:08.305354 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.305363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.305405 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.305838 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.305839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.305842 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.305887 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.305888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306787 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.306787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.307566 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.307568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.308180 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.308247 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.308252 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.308333 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.308383 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260925 10:03:08.308732 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.19.46.194:33811)
0925 10:03:08.305709 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.305711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.305719 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:08.305743 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.305744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.305755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.305957 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.305997 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306705 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.307653 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.308077 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308177 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.308191 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.308587 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.308647 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
I20260925 10:03:08.308975 26780 tablet_service.cc:1467] Processing CreateTablet for tablet ff45acdd639144b9a0b14e417023b8af (DEFAULT_TABLE table=test-table [id=9eab9347e88b4abc80f3153f5c5ed08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:08.309036 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:40869 (local address 127.0.0.1:60832)
0925 10:03:08.305411 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.305420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.305481 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.305958 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.305959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.305961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.306002 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.306003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306685 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.306686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.307965 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.307967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.308734 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.308847 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.308852 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.308912 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.308956 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":86,"thread_start_us":88,"threads_started":1}
I20260925 10:03:08.309064 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.19.46.195:40869)
0925 10:03:08.305751 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.305753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.305761 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:08.305789 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.305790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.305802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.305864 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.306097 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306595 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308077 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.308627 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308688 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.308700 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.308934 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.308984 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260925 10:03:08.309120 26780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45acdd639144b9a0b14e417023b8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:08.309288 26916 tablet_service.cc:1467] Processing CreateTablet for tablet ff45acdd639144b9a0b14e417023b8af (DEFAULT_TABLE table=test-table [id=9eab9347e88b4abc80f3153f5c5ed08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:08.309387 26916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45acdd639144b9a0b14e417023b8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:08.310428 27069 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Bootstrap starting.
I20260925 10:03:08.310740 27070 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Bootstrap starting.
I20260925 10:03:08.310832 27069 tablet_bootstrap.cc:654] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:08.311126 27070 tablet_bootstrap.cc:654] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:08.311444 27069 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: No bootstrap required, opened a new log
I20260925 10:03:08.311491 27069 ts_tablet_manager.cc:1397] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:08.311614 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:37819 (local address 127.0.0.1:43566)
0925 10:03:08.305242 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.305254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.305291 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.305578 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.305578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.305580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.305635 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.305637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306673 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.306674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.308246 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.308248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.311357 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.311358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.311422 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.311428 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.311482 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.311535 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":374}
I20260925 10:03:08.311641 27069 raft_consensus.cc:348] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.311735 27070 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: No bootstrap required, opened a new log
I20260925 10:03:08.311756 27069 raft_consensus.cc:374] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:08.311784 27069 raft_consensus.cc:729] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813640b15f584c4e94b1fdc9d52ee31c, State: Initialized, Role: FOLLOWER
I20260925 10:03:08.311776 27070 ts_tablet_manager.cc:1397] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:08.311873 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.305330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.19.46.193:37819)
0925 10:03:08.305428 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.305430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.305460 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:03:08.305488 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.305489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.305501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.305566 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.305725 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.306577 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.308329 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.311229 (+ 2900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.311348 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.311364 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.311737 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.311796 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260925 10:03:08.311867 27069 consensus_queue.cc:260] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.311894 27070 raft_consensus.cc:348] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.311962 27070 raft_consensus.cc:374] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:08.311985 27070 raft_consensus.cc:729] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd72b40b5ed4a15975b5e46061b0747, State: Initialized, Role: FOLLOWER
I20260925 10:03:08.312032 27070 consensus_queue.cc:260] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.312079 27069 ts_tablet_manager.cc:1428] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:08.312096 26643 tablet_service.cc:1467] Processing CreateTablet for tablet ff45acdd639144b9a0b14e417023b8af (DEFAULT_TABLE table=test-table [id=9eab9347e88b4abc80f3153f5c5ed08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:08.312177 27070 ts_tablet_manager.cc:1428] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:08.312196 26643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45acdd639144b9a0b14e417023b8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:08.312774 26982 heartbeater.cc:503] Master 127.19.46.252:44641 was elected leader, sending a full tablet report...
I20260925 10:03:08.312969 26846 heartbeater.cc:503] Master 127.19.46.252:44641 was elected leader, sending a full tablet report...
I20260925 10:03:08.313827 27075 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Bootstrap starting.
I20260925 10:03:08.314239 27075 tablet_bootstrap.cc:654] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:08.314337 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.313854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:58289)
0925 10:03:08.314253 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.314290 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":88,"threads_started":1}
I20260925 10:03:08.314610 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.314123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:50199)
0925 10:03:08.314531 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.314567 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":74,"threads_started":1}
I20260925 10:03:08.314711 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.314550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:32807)
0925 10:03:08.314614 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.314650 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:03:08.314810 27075 tablet_bootstrap.cc:492] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: No bootstrap required, opened a new log
I20260925 10:03:08.314858 27075 ts_tablet_manager.cc:1397] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:08.314958 27075 raft_consensus.cc:348] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.315019 27075 raft_consensus.cc:374] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:08.315033 27075 raft_consensus.cc:729] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0f9049117147d1ab05ffc5627e0df1, State: Initialized, Role: FOLLOWER
I20260925 10:03:08.315065 27075 consensus_queue.cc:260] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.315198 27075 ts_tablet_manager.cc:1428] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:08.315608 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.315394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:58281)
0925 10:03:08.315509 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.315568 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":77,"threads_started":1}
I20260925 10:03:08.315865 26709 heartbeater.cc:503] Master 127.19.46.252:44641 was elected leader, sending a full tablet report...
W20260925 10:03:08.348479 26712 tablet.cc:2369] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:03:08.401620 26985 tablet.cc:2369] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:03:08.491178 26849 tablet.cc:2369] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:03:08.528582 27073 raft_consensus.cc:482] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:08.528698 27073 raft_consensus.cc:504] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.529122 27073 leader_election.cc:288] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819), fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:08.532850 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.529347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:40869 (local address 127.19.46.194:57587)
0925 10:03:08.529437 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.529467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.529529 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.529807 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.529808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.529811 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.529913 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.529915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.531006 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.531007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.531900 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.531903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.532532 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.532620 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.532625 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.532712 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.532762 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260925 10:03:08.532850 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.529426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57587 (local address 127.19.46.195:40869)
0925 10:03:08.529500 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.529502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.529513 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:08.529686 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.529688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.529703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.529785 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.530001 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.530845 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532024 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.532427 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.532543 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.532735 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.532775 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:03:08.532994 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.529494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:38757 (local address 127.19.46.193:37819)
0925 10:03:08.529544 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.529546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.529694 (+ 148us) server_negotiation.cc:407] Connection header received
0925 10:03:08.529727 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.529728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.529738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.529799 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:08.530001 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.530838 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532032 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.532497 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532629 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.532641 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.532877 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.532927 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:03:08.532994 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.529389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:37819 (local address 127.19.46.194:38757)
0925 10:03:08.529605 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.529616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:08.529680 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:08.529825 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:08.529825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:08.529827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:08.529914 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.529915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.530961 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.530963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.531918 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:08.531921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.532657 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.532658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:08.532741 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.532745 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:08.532853 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:03:08.532901 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":26,"thread_start_us":160,"threads_started":1}
I20260925 10:03:08.533416 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45acdd639144b9a0b14e417023b8af" candidate_uuid: "813640b15f584c4e94b1fdc9d52ee31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd72b40b5ed4a15975b5e46061b0747" is_pre_election: true
I20260925 10:03:08.533561 26936 raft_consensus.cc:2454] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813640b15f584c4e94b1fdc9d52ee31c in term 0.
I20260925 10:03:08.533494 26663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45acdd639144b9a0b14e417023b8af" candidate_uuid: "813640b15f584c4e94b1fdc9d52ee31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d0f9049117147d1ab05ffc5627e0df1" is_pre_election: true
I20260925 10:03:08.533675 26663 raft_consensus.cc:2454] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813640b15f584c4e94b1fdc9d52ee31c in term 0.
I20260925 10:03:08.533769 26736 leader_election.cc:302] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813640b15f584c4e94b1fdc9d52ee31c, fbd72b40b5ed4a15975b5e46061b0747; no voters:
I20260925 10:03:08.533921 27073 raft_consensus.cc:2781] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:08.533972 27073 raft_consensus.cc:482] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:08.534024 27073 raft_consensus.cc:3037] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:08.534782 27073 raft_consensus.cc:504] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.534960 27073 leader_election.cc:288] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [CANDIDATE]: Term 1 election: Requested vote from peers 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819), fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869)
I20260925 10:03:08.535153 26663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45acdd639144b9a0b14e417023b8af" candidate_uuid: "813640b15f584c4e94b1fdc9d52ee31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d0f9049117147d1ab05ffc5627e0df1"
I20260925 10:03:08.535192 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45acdd639144b9a0b14e417023b8af" candidate_uuid: "813640b15f584c4e94b1fdc9d52ee31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd72b40b5ed4a15975b5e46061b0747"
I20260925 10:03:08.535271 26936 raft_consensus.cc:3037] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:08.535288 26663 raft_consensus.cc:3037] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:08.535908 26936 raft_consensus.cc:2454] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813640b15f584c4e94b1fdc9d52ee31c in term 1.
I20260925 10:03:08.536109 26736 leader_election.cc:302] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813640b15f584c4e94b1fdc9d52ee31c, fbd72b40b5ed4a15975b5e46061b0747; no voters:
I20260925 10:03:08.536145 26663 raft_consensus.cc:2454] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813640b15f584c4e94b1fdc9d52ee31c in term 1.
I20260925 10:03:08.536222 27073 raft_consensus.cc:2781] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:08.536423 27073 raft_consensus.cc:686] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 1 LEADER]: Becoming Leader. State: Replica: 813640b15f584c4e94b1fdc9d52ee31c, State: Running, Role: LEADER
I20260925 10:03:08.536513 27073 consensus_queue.cc:237] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } }
I20260925 10:03:08.536864 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.536713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:55915)
0925 10:03:08.536758 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.536836 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 10:03:08.537415 26529 catalog_manager.cc:5495] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194). New cstate: current_term: 1 leader_uuid: "813640b15f584c4e94b1fdc9d52ee31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813640b15f584c4e94b1fdc9d52ee31c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 33811 } health_report { overall_health: HEALTHY } } }
I20260925 10:03:08.596132 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.593327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.19.46.194:33811)
0925 10:03:08.593409 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:08.593411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:08.593420 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:08.593504 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:08.593506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:08.593531 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:08.593614 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:08.593789 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.594300 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.595262 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:08.595460 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:08.595528 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:08.595725 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:08.595834 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:08.595914 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:08.596043 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:03:08.596067 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:03:08.597800 26936 raft_consensus.cc:1261] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Refusing update from remote peer 813640b15f584c4e94b1fdc9d52ee31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:08.597898 26663 raft_consensus.cc:1261] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Refusing update from remote peer 813640b15f584c4e94b1fdc9d52ee31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:08.598269 27073 consensus_queue.cc:1035] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd72b40b5ed4a15975b5e46061b0747" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 40869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:08.598464 27082 consensus_queue.cc:1035] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d0f9049117147d1ab05ffc5627e0df1" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:08.599037 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.598908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:51849)
0925 10:03:08.598949 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.598995 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 10:03:08.599956 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.599824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:59281)
0925 10:03:08.599884 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.599920 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:03:08.600646 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.600543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.194:42569)
0925 10:03:08.600586 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.600613 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 10:03:08.600780 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.600689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.193:47219)
0925 10:03:08.600727 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.600751 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 10:03:08.602798 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.602684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:44019 (local address 127.19.46.195:49009)
0925 10:03:08.602725 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.602758 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 10:03:08.603147 26530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:08.603267 26530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:08.604722 26530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c195dab1cf947f3b3bfd9141c796c04: Sending DeleteTablet for 3 replicas of tablet ff45acdd639144b9a0b14e417023b8af
I20260925 10:03:08.605082 26643 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45acdd639144b9a0b14e417023b8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:08 UTC) from {username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:43566
I20260925 10:03:08.605094 26780 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45acdd639144b9a0b14e417023b8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:08 UTC) from {username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:43340
I20260925 10:03:08.605283 27096 tablet_replica.cc:330] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: stopping tablet replica
I20260925 10:03:08.605341 27096 raft_consensus.cc:2229] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:08.605372 26916 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45acdd639144b9a0b14e417023b8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:08 UTC) from {username='oryx', principal='oryx/127.19.46.252@KRBTEST.COM'} at 127.0.0.1:60832
I20260925 10:03:08.605425 27096 raft_consensus.cc:2258] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:08.605525 27097 tablet_replica.cc:330] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: stopping tablet replica
I20260925 10:03:08.605575 27097 raft_consensus.cc:2229] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:08.605659 27097 raft_consensus.cc:2258] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:08.605883 27096 ts_tablet_manager.cc:1905] T ff45acdd639144b9a0b14e417023b8af P 5d0f9049117147d1ab05ffc5627e0df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:08.605909 27097 ts_tablet_manager.cc:1905] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:08.605914 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26490
I20260925 10:03:08.607699 27097 ts_tablet_manager.cc:1918] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:08.607758 27097 log.cc:1196] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/ff45acdd639144b9a0b14e417023b8af
I20260925 10:03:08.607954 27097 ts_tablet_manager.cc:1939] T ff45acdd639144b9a0b14e417023b8af P 813640b15f584c4e94b1fdc9d52ee31c: Deleting consensus metadata
I20260925 10:03:08.608332 26514 catalog_manager.cc:4841] TS 813640b15f584c4e94b1fdc9d52ee31c (127.19.46.194:33811): tablet ff45acdd639144b9a0b14e417023b8af (table test-table [id=9eab9347e88b4abc80f3153f5c5ed08c]) successfully deleted
I20260925 10:03:08.613570 27098 tablet_replica.cc:330] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: stopping tablet replica
I20260925 10:03:08.613664 27098 raft_consensus.cc:2229] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:08.613718 27098 raft_consensus.cc:2258] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:08.614779 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:08.614609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:37819 (local address 127.0.0.1:43580)
0925 10:03:08.614676 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:08.614706 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:03:08.614955 27098 ts_tablet_manager.cc:1905] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 10:03:08.615018 26515 catalog_manager.cc:4584] TS 5d0f9049117147d1ab05ffc5627e0df1 (127.19.46.193:37819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff45acdd639144b9a0b14e417023b8af: Network error: Client connection negotiation failed: client connection to 127.19.46.193:37819: connect: Connection refused (error 111)
I20260925 10:03:08.616892 27098 ts_tablet_manager.cc:1918] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:08.616997 27098 log.cc:1196] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/ff45acdd639144b9a0b14e417023b8af
I20260925 10:03:08.617138 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26717
I20260925 10:03:08.617832 27098 ts_tablet_manager.cc:1939] T ff45acdd639144b9a0b14e417023b8af P fbd72b40b5ed4a15975b5e46061b0747: Deleting consensus metadata
I20260925 10:03:08.618242 26517 catalog_manager.cc:4841] TS fbd72b40b5ed4a15975b5e46061b0747 (127.19.46.195:40869): tablet ff45acdd639144b9a0b14e417023b8af (table test-table [id=9eab9347e88b4abc80f3153f5c5ed08c]) successfully deleted
I20260925 10:03:08.622889 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26853
I20260925 10:03:08.627750 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26210
I20260925 10:03:08.632930 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 26397
2026-09-25T10:03:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:08 dist-test-slave-f9rl krb5kdc[27101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:08 dist-test-slave-f9rl krb5kdc[27101](info): set up 2 sockets
Sep 25 10:03:08 dist-test-slave-f9rl krb5kdc[27101](info): commencing operation
krb5kdc: starting...
W20260925 10:03:09.719389 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:09 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab.
Entry for principal oryx/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab.
I20260925 10:03:09.848676 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37713
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab
--principal=oryx/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:09.931826 27114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:09.932039 27114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:09.932062 27114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:09.933192 27114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:09.933233 27114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:03:09.933245 27114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:09.933259 27114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:09.933270 27114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:09.933283 27114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:09.934860 27114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37713
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.254.keytab
--principal=oryx/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:09.935137 27114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:09.935369 27114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:09.940483 27120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:09.940600 27114 server_base.cc:1034] running on GCE node
W20260925 10:03:09.940487 27119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:09.940483 27122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:09.940913 27114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:09.941200 27114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:09.942397 27114 hybrid_clock.cc:648] HybridClock initialized: now 1790330589942390 us; error 37 us; skew 500 ppm
Sep 25 10:03:09 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330589, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:09.945042 27114 init.cc:377] Logged in from keytab as oryx/127.19.46.254@KRBTEST.COM (short username oryx)
I20260925 10:03:09.945551 27114 webserver.cc:460] Webserver started at http://127.19.46.254:42981/ using document root <none> and password file <none>
I20260925 10:03:09.945811 27114 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:09.945878 27114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:09.946023 27114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:09.947095 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16"
format_stamp: "Formatted at 2026-09-25 10:03:09 on dist-test-slave-f9rl"
server_key: "e2f4693b060b9dbb72bd0f0591c56b0e"
server_key_iv: "22a682df78cb5ed0c2adcafc148a807f"
server_key_version: "encryptionkey@0"
I20260925 10:03:09.947456 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16"
format_stamp: "Formatted at 2026-09-25 10:03:09 on dist-test-slave-f9rl"
server_key: "e2f4693b060b9dbb72bd0f0591c56b0e"
server_key_iv: "22a682df78cb5ed0c2adcafc148a807f"
server_key_version: "encryptionkey@0"
I20260925 10:03:09.948827 27114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:09.949734 27129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:09.949975 27114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:09.950098 27114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16"
format_stamp: "Formatted at 2026-09-25 10:03:09 on dist-test-slave-f9rl"
server_key: "e2f4693b060b9dbb72bd0f0591c56b0e"
server_key_iv: "22a682df78cb5ed0c2adcafc148a807f"
server_key_version: "encryptionkey@0"
I20260925 10:03:09.950201 27114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:09.973261 27114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:09.975667 27114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:09.975841 27114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:09.980396 27114 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:37713
I20260925 10:03:09.980496 27191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:37713 every 8 connection(s)
I20260925 10:03:09.980804 27114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:09.981941 27192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:09.984153 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27114
I20260925 10:03:09.984300 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:09.984390 27192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Bootstrap starting.
I20260925 10:03:09.984521 19643 external_mini_cluster.cc:1428] Setting key c8de43112c21b7915897252fbbef4124
I20260925 10:03:09.985327 27192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:09.985667 27192 log.cc:824] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:09.986352 27192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: No bootstrap required, opened a new log
I20260925 10:03:09.988204 27192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } }
I20260925 10:03:09.988376 27192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:09.988426 27192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ca90ad2a6345e59e2b2d71fbb6bc16, State: Initialized, Role: FOLLOWER
I20260925 10:03:09.988590 27192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } }
I20260925 10:03:09.988679 27192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:09.988723 27192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:09.988771 27192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:09.989568 27192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } }
I20260925 10:03:09.989691 27192 leader_election.cc:302] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ca90ad2a6345e59e2b2d71fbb6bc16; no voters:
I20260925 10:03:09.989886 27192 leader_election.cc:288] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:09.989954 27197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:09.990126 27197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [term 1 LEADER]: Becoming Leader. State: Replica: 66ca90ad2a6345e59e2b2d71fbb6bc16, State: Running, Role: LEADER
I20260925 10:03:09.990188 27192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:09.990247 27197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } }
I20260925 10:03:09.990588 27197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ca90ad2a6345e59e2b2d71fbb6bc16. Latest consensus state: current_term: 1 leader_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } } }
I20260925 10:03:09.990682 27197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:09.990598 27198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ca90ad2a6345e59e2b2d71fbb6bc16" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37713 } } }
I20260925 10:03:09.990840 27198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:09.991001 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 10:03:09 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:09.991789 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:09.993842 27203 catalog_manager.cc:1347] Generated new cluster ID: c3459e9ae7dc4470b5e725aa3fbc5792
I20260925 10:03:09.993908 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:09.995456 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:09.985015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.19.46.254:37713)
0925 10:03:09.985177 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:09.985185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:09.985216 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:03:09.985717 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:09.985725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:09.985795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:09.985961 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:09.986596 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:09.987501 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:09.988251 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:09.988743 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:09.988868 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:09.991871 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:09.991875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:09.991876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:09.991888 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:09.993944 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:09.994313 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:09.994316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:09.994317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:09.994410 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:09.994613 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:09.994614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:09.994615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:09.994858 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:09.995017 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:09.995140 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:03:09.995275 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260925 10:03:10.025616 27203 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:10.026404 27203 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:10.031509 27203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Generated new TSK 0
I20260925 10:03:10.031713 27203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
I20260925 10:03:10.060184 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:10.149055 27218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:10.149281 27218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:10.149313 27218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:10.150631 27218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:10.150682 27218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:10.150704 27218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:03:10.150722 27218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:10.152267 27218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:10.152513 27218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:10.152745 27218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:10.158372 27226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:10.158438 27224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:10.158447 27223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:10.158587 27218 server_base.cc:1034] running on GCE node
I20260925 10:03:10.158854 27218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:10.159142 27218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:10.160392 27218 hybrid_clock.cc:648] HybridClock initialized: now 1790330590160371 us; error 52 us; skew 500 ppm
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:10.163573 27218 init.cc:377] Logged in from keytab as oryx/127.19.46.193@KRBTEST.COM (short username oryx)
I20260925 10:03:10.164086 27218 webserver.cc:460] Webserver started at http://127.19.46.193:37971/ using document root <none> and password file <none>
I20260925 10:03:10.164328 27218 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:10.164395 27218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:10.164546 27218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:10.165743 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "32dc1d73b4e343d7b386e1097068db2c"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "0d8de97e2f2a93481fc30bab8c7da645"
server_key_iv: "68778e628067c1b04b1cc4da6d50f51d"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.166113 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "32dc1d73b4e343d7b386e1097068db2c"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "0d8de97e2f2a93481fc30bab8c7da645"
server_key_iv: "68778e628067c1b04b1cc4da6d50f51d"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.167562 27218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.168422 27233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.168630 27218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.168731 27218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "32dc1d73b4e343d7b386e1097068db2c"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "0d8de97e2f2a93481fc30bab8c7da645"
server_key_iv: "68778e628067c1b04b1cc4da6d50f51d"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.168840 27218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:10.180169 27218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:10.182442 27218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:10.182612 27218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:10.183158 27218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:10.183220 27218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.183254 27218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:10.183283 27218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:10.190819 27218 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:34741
I20260925 10:03:10.191027 27349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:34741 every 8 connection(s)
I20260925 10:03:10.191273 27218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:10.194057 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.183652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:48895 (local address 127.19.46.254:37713)
0925 10:03:10.183721 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.183724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.183982 (+ 258us) server_negotiation.cc:407] Connection header received
0925 10:03:10.184751 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.184752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.184763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.184812 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.185819 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.186413 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.188093 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.188315 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.188400 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.191070 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.191071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.191072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.191080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.192709 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.193194 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.193195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.193195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.193248 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.193485 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.193486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.193486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.193603 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.193655 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.193914 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.193990 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:03:10.194156 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.183621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.193:48895)
0925 10:03:10.183820 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.183874 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.184642 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.184891 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.184896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.185333 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.185712 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.185720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.186509 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.186510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.187998 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.188001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.188422 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.188423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.188548 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.189097 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.189105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.190944 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.192791 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.192797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.192799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.193118 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.193300 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.193300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.193301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.193398 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.193646 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.193649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.193765 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.193839 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.193965 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260925 10:03:10.194618 27350 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37713
I20260925 10:03:10.194702 27350 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:10.194892 27350 heartbeater.cc:511] Master 127.19.46.254:37713 requested a full tablet report, sending...
I20260925 10:03:10.195475 27145 ts_manager.cc:194] Registered new tserver with Master: 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741)
I20260925 10:03:10.196147 27145 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:48895
I20260925 10:03:10.196409 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27218
I20260925 10:03:10.196520 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:10.196750 19643 external_mini_cluster.cc:1428] Setting key 27a7c3540500b96235e92181a6578c6f
I20260925 10:03:10.202509 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.195698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:36255 (local address 127.19.46.254:37713)
0925 10:03:10.195744 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.195746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.195809 (+ 63us) server_negotiation.cc:407] Connection header received
0925 10:03:10.195887 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.195888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.195898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.195946 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.196943 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.197517 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.198533 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.198725 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.198788 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.199773 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.199775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.199775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.199782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.201253 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.201760 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.201761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.201761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.201812 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.202033 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.202033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.202033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.202133 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.202183 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.202369 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.202444 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:03:10.202648 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.195639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.193:36255)
0925 10:03:10.195757 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.195768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.195824 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.195947 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.195947 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.196131 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.196181 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.196183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.197604 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.197606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.198429 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.198431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.198801 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.198802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.198871 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.199410 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.199414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.199689 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.201350 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.201351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.201352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.201670 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.201879 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.201880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.201880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.201962 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.202201 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.202202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.202261 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.202356 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.202504 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
I20260925 10:03:10.225718 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:10.314540 27359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:10.314762 27359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:10.314787 27359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:10.315986 27359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:10.316032 27359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:10.316054 27359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:03:10.316071 27359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:10.317610 27359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:10.317821 27359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:10.318012 27359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:10.323241 27367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:10.323431 27359 server_base.cc:1034] running on GCE node
W20260925 10:03:10.323226 27364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:10.323230 27365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:10.323762 27359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:10.323997 27359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:10.325163 27359 hybrid_clock.cc:648] HybridClock initialized: now 1790330590325125 us; error 60 us; skew 500 ppm
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:10.328295 27359 init.cc:377] Logged in from keytab as oryx/127.19.46.194@KRBTEST.COM (short username oryx)
I20260925 10:03:10.328827 27359 webserver.cc:460] Webserver started at http://127.19.46.194:40807/ using document root <none> and password file <none>
I20260925 10:03:10.329099 27359 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:10.329169 27359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:10.329313 27359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:10.330472 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "c0468a4057c94186abfa8d9f16a7869d"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "43796a7aa1e023669961fb9ebf964684"
server_key_iv: "0bf5ef957687328a05162827a98493d6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.330830 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "c0468a4057c94186abfa8d9f16a7869d"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "43796a7aa1e023669961fb9ebf964684"
server_key_iv: "0bf5ef957687328a05162827a98493d6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.332202 27359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:03:10.332979 27374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.333169 27359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.333266 27359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "c0468a4057c94186abfa8d9f16a7869d"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "43796a7aa1e023669961fb9ebf964684"
server_key_iv: "0bf5ef957687328a05162827a98493d6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.333344 27359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:10.346556 27359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:10.348606 27359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:10.348775 27359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:10.349401 27359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:10.349474 27359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.349511 27359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:10.349558 27359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:10.357013 27359 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37801
I20260925 10:03:10.357074 27490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37801 every 8 connection(s)
I20260925 10:03:10.357386 27359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:10.359838 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.349798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:36217 (local address 127.19.46.254:37713)
0925 10:03:10.349864 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.349868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.350127 (+ 259us) server_negotiation.cc:407] Connection header received
0925 10:03:10.350643 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.350645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.350658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.350739 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.351512 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.352346 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.353550 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.353896 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.353974 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.357010 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.357012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.357012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.357026 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.358568 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.358989 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.358989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.358990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.359043 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.359247 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.359247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.359248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.359355 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.359422 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.359706 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.359770 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:03:10.360124 27383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.349809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.194:36217)
0925 10:03:10.349982 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.350025 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.350543 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.350842 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.350847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.351182 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.351420 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.351426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.352456 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.352457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.353422 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.353425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.353995 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.353996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.354141 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.354680 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.354691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.356896 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.358671 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.358678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.358680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.358915 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.359097 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.359097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.359098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.359196 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.359447 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.359453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.359581 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.359682 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.359828 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":130,"threads_started":1}
I20260925 10:03:10.360651 27491 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37713
I20260925 10:03:10.360740 27491 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:10.360970 27491 heartbeater.cc:511] Master 127.19.46.254:37713 requested a full tablet report, sending...
I20260925 10:03:10.361356 27145 ts_manager.cc:194] Registered new tserver with Master: c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801)
I20260925 10:03:10.361653 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27359
I20260925 10:03:10.361752 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:10.361948 19643 external_mini_cluster.cc:1428] Setting key 695340508bca094cb34bd1b495bc6cae
I20260925 10:03:10.361923 27145 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:36217
I20260925 10:03:10.368778 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.361739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:38915 (local address 127.19.46.254:37713)
0925 10:03:10.361791 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.361793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.361992 (+ 199us) server_negotiation.cc:407] Connection header received
0925 10:03:10.362309 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.362310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.362321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.362376 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.362769 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.363500 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.364387 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.364620 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.364724 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.366050 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.366052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.366052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.366059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.367574 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.368003 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.368004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.368004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.368056 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.368268 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.368269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.368269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.368416 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.368478 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.368647 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.368711 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:03:10.368885 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.361690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.194:38915)
0925 10:03:10.361918 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.361928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.362059 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.362401 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.362402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.362618 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.362686 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.362689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.363632 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.363633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.364242 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.364244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.364726 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.364727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.364799 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.365349 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.365363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.365939 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.367666 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.367667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.367668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.367924 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.368121 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.368122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.368123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.368199 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.368482 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.368483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.368540 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.368628 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.368756 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
I20260925 10:03:10.389209 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:10.470709 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:10.471076 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:10.471125 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:10.472558 27500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:10.472625 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:10.472667 27500 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:03:10.472723 27500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:10.474509 27500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.46.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:10.474817 27500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:10.475073 27500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:10.480702 27505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:10.480858 27500 server_base.cc:1034] running on GCE node
W20260925 10:03:10.480682 27506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:10.480676 27508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:10.481185 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:10.481424 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:10.482637 27500 hybrid_clock.cc:648] HybridClock initialized: now 1790330590482622 us; error 31 us; skew 500 ppm
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:10.485476 27500 init.cc:377] Logged in from keytab as oryx/127.19.46.195@KRBTEST.COM (short username oryx)
I20260925 10:03:10.485986 27500 webserver.cc:460] Webserver started at http://127.19.46.195:43685/ using document root <none> and password file <none>
I20260925 10:03:10.486232 27500 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:10.486305 27500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:10.486497 27500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:10.487665 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "dcbbebc5878da69edf42da11e25f360a"
server_key_iv: "a250061782a08ea25b1b7efca50b0cf6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.488057 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "dcbbebc5878da69edf42da11e25f360a"
server_key_iv: "a250061782a08ea25b1b7efca50b0cf6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.489598 27500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:03:10.490459 27515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.490654 27500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.490780 27500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
format_stamp: "Formatted at 2026-09-25 10:03:10 on dist-test-slave-f9rl"
server_key: "dcbbebc5878da69edf42da11e25f360a"
server_key_iv: "a250061782a08ea25b1b7efca50b0cf6"
server_key_version: "encryptionkey@0"
I20260925 10:03:10.490875 27500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:10.501348 27500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:10.503660 27500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:10.503830 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:10.504423 27500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:10.504495 27500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.504539 27500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:10.504567 27500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:10.512171 27500 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:46581
I20260925 10:03:10.512261 27631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:46581 every 8 connection(s)
I20260925 10:03:10.512624 27500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:10.514793 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.504868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:53037 (local address 127.19.46.254:37713)
0925 10:03:10.504945 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.504948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.505185 (+ 237us) server_negotiation.cc:407] Connection header received
0925 10:03:10.505746 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.505748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.505762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.505848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.506518 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.507377 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.508547 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.508922 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.509003 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.511527 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.511529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.511530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.511540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.513231 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.513656 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.513657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.513657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.513720 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.514099 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.514100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.514101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.514229 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.514291 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.514642 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.514725 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:03:10.514700 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27500
I20260925 10:03:10.514917 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:10.515082 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.504864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.195:53037)
0925 10:03:10.505044 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.505087 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.505640 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.505936 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.505939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.506219 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.506412 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.506416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.507499 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.507511 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.508458 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.508460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.509029 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.509030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.509166 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.509761 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.509782 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.511423 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.513302 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.513307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.513309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.513588 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.513835 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.513836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.513837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.513950 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.514324 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.514330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.514460 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.514613 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.514818 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":115,"threads_started":1}
I20260925 10:03:10.515152 19643 external_mini_cluster.cc:1428] Setting key f691c1efada78cb4f568f03bc8751c20
I20260925 10:03:10.515652 27632 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37713
I20260925 10:03:10.515748 27632 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:10.515964 27632 heartbeater.cc:511] Master 127.19.46.254:37713 requested a full tablet report, sending...
I20260925 10:03:10.516356 27146 ts_manager.cc:194] Registered new tserver with Master: e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:10.516822 27146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:53037
I20260925 10:03:10.522786 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.516696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:37655 (local address 127.19.46.254:37713)
0925 10:03:10.516730 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.516732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.516853 (+ 121us) server_negotiation.cc:407] Connection header received
0925 10:03:10.516897 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.516898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.516908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.516966 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.517470 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.517954 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.518652 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.518863 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.518937 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.520164 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.520165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.520166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.520173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.521813 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.522160 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.522161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.522161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.522214 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.522374 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.522375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.522375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.522480 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.522531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.522652 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.522714 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:03:10.522807 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.516637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.19.46.195:37655)
0925 10:03:10.516772 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.516784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.516847 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.517081 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.517082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.517302 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.517370 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.517372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.518048 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.518049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.518571 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.518572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.518936 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.518936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.519004 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.519514 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.519520 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.520062 (+ 542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.521894 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.521895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.521896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.522075 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.522271 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.522272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.522272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.522326 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.522528 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.522529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.522580 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.522632 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.522716 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260925 10:03:10.526856 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:03:10.548988 27195 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:10.543172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.19.46.254:37713)
0925 10:03:10.543240 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.543242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.543253 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:10.543306 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.543307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.543323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.543387 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.544000 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.544506 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.545285 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.545563 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.545653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.548876 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41738: BlockingRecv error: recv got EOF from 127.0.0.1:41738 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:10.557346 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.549850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.19.46.254:37713)
0925 10:03:10.549936 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.549939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.550029 (+ 90us) server_negotiation.cc:407] Connection header received
0925 10:03:10.550121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.550122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.550133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.550182 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.550657 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.551351 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.552099 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.552322 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.552425 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.554775 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.554776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.554776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.554785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.556157 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.556510 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.556511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.556512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.556568 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.556758 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.556759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.556759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.556876 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.556980 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.557093 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.557171 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:03:10.559365 27146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:10.561277 27146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:10.568522 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.564165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.19.46.194:37801)
0925 10:03:10.564708 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.564712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.564720 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:10.564756 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.564758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.564773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.564844 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.565044 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.566322 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.567540 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.568048 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.568135 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.568219 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.568374 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.568420 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":90,"threads_started":1}
I20260925 10:03:10.568500 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.563948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.0.0.1:44112)
0925 10:03:10.564290 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.564338 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.564427 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.564668 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.564671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.564673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.564761 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.564762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.565903 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.565904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.567095 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.567098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.568169 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.568170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.568243 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.568272 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.568310 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.568369 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":78,"threads_started":1}
I20260925 10:03:10.568660 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.564032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37801 (local address 127.0.0.1:57730)
0925 10:03:10.564341 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.564352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.564425 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.564917 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.564918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.564923 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.564977 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.564979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.566440 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.566442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.567442 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.567444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.568147 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.568148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.568243 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.568263 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.568352 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.568405 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":33,"thread_start_us":95,"threads_started":1}
I20260925 10:03:10.568770 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.564112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.19.46.193:34741)
0925 10:03:10.564257 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.564263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.564415 (+ 152us) server_negotiation.cc:407] Connection header received
0925 10:03:10.564513 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.564516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.564534 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.564842 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.564899 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.565795 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.567188 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.568065 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.568460 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.568550 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.568581 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.568631 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260925 10:03:10.569600 27283 tablet_service.cc:1467] Processing CreateTablet for tablet b99b39a2b67d4ea7975a23269766a082 (DEFAULT_TABLE table=test-table [id=97a46572c47544b5864f20d27e7f04c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.569599 27425 tablet_service.cc:1467] Processing CreateTablet for tablet b99b39a2b67d4ea7975a23269766a082 (DEFAULT_TABLE table=test-table [id=97a46572c47544b5864f20d27e7f04c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.569993 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99b39a2b67d4ea7975a23269766a082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.570039 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99b39a2b67d4ea7975a23269766a082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.573480 27660 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Bootstrap starting.
I20260925 10:03:10.573530 27661 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Bootstrap starting.
I20260925 10:03:10.574172 27660 tablet_bootstrap.cc:654] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.574215 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.564182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:46581 (local address 127.0.0.1:46704)
0925 10:03:10.564541 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.564551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.564590 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.564860 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.564861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.564864 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.564925 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.564927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.566453 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.566454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.567734 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.567737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.573891 (+ 6154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.573892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.573977 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.573983 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.574052 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.574119 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":69,"threads_started":1}
I20260925 10:03:10.574232 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.564401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.19.46.195:46581)
0925 10:03:10.564590 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.564609 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.564620 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:10.564685 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.564687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.564704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.564783 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.565014 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.566346 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.567844 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.573774 (+ 5930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.573867 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.573985 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.574067 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.574126 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260925 10:03:10.574536 27660 log.cc:824] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:10.574672 27661 tablet_bootstrap.cc:654] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.575076 27661 log.cc:824] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:10.575011 27566 tablet_service.cc:1467] Processing CreateTablet for tablet b99b39a2b67d4ea7975a23269766a082 (DEFAULT_TABLE table=test-table [id=97a46572c47544b5864f20d27e7f04c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.575344 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99b39a2b67d4ea7975a23269766a082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.575423 27660 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: No bootstrap required, opened a new log
I20260925 10:03:10.575486 27660 ts_tablet_manager.cc:1397] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:03:10.575990 27661 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: No bootstrap required, opened a new log
I20260925 10:03:10.576076 27661 ts_tablet_manager.cc:1397] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:03:10.577668 27661 raft_consensus.cc:348] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.577641 27660 raft_consensus.cc:348] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.577826 27660 raft_consensus.cc:374] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.577827 27661 raft_consensus.cc:374] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.577874 27660 raft_consensus.cc:729] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.577874 27661 raft_consensus.cc:729] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.578013 27660 consensus_queue.cc:260] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.578024 27661 consensus_queue.cc:260] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.578311 27661 ts_tablet_manager.cc:1428] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:03:10.578310 27660 ts_tablet_manager.cc:1428] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:10.578450 27350 heartbeater.cc:503] Master 127.19.46.254:37713 was elected leader, sending a full tablet report...
I20260925 10:03:10.578467 27491 heartbeater.cc:503] Master 127.19.46.254:37713 was elected leader, sending a full tablet report...
I20260925 10:03:10.579200 27667 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Bootstrap starting.
I20260925 10:03:10.580060 27667 tablet_bootstrap.cc:654] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.580404 27667 log.cc:824] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:10.581399 27667 tablet_bootstrap.cc:492] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: No bootstrap required, opened a new log
I20260925 10:03:10.581707 27667 ts_tablet_manager.cc:1397] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:03:10.583460 27667 raft_consensus.cc:348] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.583611 27667 raft_consensus.cc:374] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.583634 27667 raft_consensus.cc:729] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6b0c9de504aed9e03d5885aca4b91, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.583781 27667 consensus_queue.cc:260] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.584064 27667 ts_tablet_manager.cc:1428] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:03:10.584120 27632 heartbeater.cc:503] Master 127.19.46.254:37713 was elected leader, sending a full tablet report...
W20260925 10:03:10.608131 27492 tablet.cc:2369] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:03:10.627341 27664 raft_consensus.cc:482] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:10.627482 27664 raft_consensus.cc:504] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.627789 27664 leader_election.cc:288] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:10.631690 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.628076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:42069 (local address 127.19.46.193:34741)
0925 10:03:10.628118 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.628121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.628133 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:10.628171 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.628172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.628183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.628272 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.628458 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.629250 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.630727 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.631314 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.631380 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.631388 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.631571 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.631610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:03:10.631690 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.628218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:33393 (local address 127.19.46.195:46581)
0925 10:03:10.628286 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.628288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.628543 (+ 255us) server_negotiation.cc:407] Connection header received
0925 10:03:10.628572 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.628574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.628583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.628653 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.628755 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.629414 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.630727 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.631285 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.631370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.631381 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.631571 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.631610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:03:10.631713 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.628128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:46581 (local address 127.19.46.194:33393)
0925 10:03:10.628482 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.628489 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.628525 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.628651 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.628652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.628653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.628700 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.628701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.629547 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.629548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.630633 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.630636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.631399 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.631399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.631489 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.631493 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.631557 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.631603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":2,"thread_start_us":126,"threads_started":1}
I20260925 10:03:10.631719 27383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.628003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.19.46.194:42069)
0925 10:03:10.628060 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.628073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.628119 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.628321 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.628322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.628324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.628391 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.628392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.629334 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.629335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.630633 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.630636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.631398 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.631399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.631490 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.631493 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.631557 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.631602 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":10}
I20260925 10:03:10.632117 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99b39a2b67d4ea7975a23269766a082" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" is_pre_election: true
I20260925 10:03:10.632314 27576 raft_consensus.cc:2454] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 0.
I20260925 10:03:10.632311 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99b39a2b67d4ea7975a23269766a082" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dc1d73b4e343d7b386e1097068db2c" is_pre_election: true
I20260925 10:03:10.632434 27303 raft_consensus.cc:2454] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 0.
I20260925 10:03:10.632562 27376 leader_election.cc:302] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0468a4057c94186abfa8d9f16a7869d, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:10.632723 27664 raft_consensus.cc:2781] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:10.632792 27664 raft_consensus.cc:482] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:10.632809 27664 raft_consensus.cc:3037] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.633921 27664 raft_consensus.cc:504] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.634070 27664 leader_election.cc:288] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 election: Requested vote from peers 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:10.634375 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99b39a2b67d4ea7975a23269766a082" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dc1d73b4e343d7b386e1097068db2c"
I20260925 10:03:10.634433 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99b39a2b67d4ea7975a23269766a082" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
I20260925 10:03:10.634466 27303 raft_consensus.cc:3037] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.634543 27576 raft_consensus.cc:3037] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.635219 27303 raft_consensus.cc:2454] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 1.
I20260925 10:03:10.635248 27576 raft_consensus.cc:2454] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 1.
I20260925 10:03:10.635425 27376 leader_election.cc:302] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0468a4057c94186abfa8d9f16a7869d, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:10.635529 27664 raft_consensus.cc:2781] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:10.635696 27664 raft_consensus.cc:686] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 1 LEADER]: Becoming Leader. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Running, Role: LEADER
I20260925 10:03:10.635821 27664 consensus_queue.cc:237] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } }
I20260925 10:03:10.636715 27146 catalog_manager.cc:5495] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0468a4057c94186abfa8d9f16a7869d (127.19.46.194). New cstate: current_term: 1 leader_uuid: "c0468a4057c94186abfa8d9f16a7869d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } health_report { overall_health: HEALTHY } } }
I20260925 10:03:10.642064 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.639726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.19.46.194:37801)
0925 10:03:10.639783 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.639785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.639793 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:10.639857 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.639858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.639873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.639932 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:10.640026 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.640482 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.641253 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.641509 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.641577 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.641708 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:10.641830 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:10.641891 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.641954 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.641973 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:03:10.643760 27303 raft_consensus.cc:1261] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Refusing update from remote peer c0468a4057c94186abfa8d9f16a7869d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:10.643759 27576 raft_consensus.cc:1261] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Refusing update from remote peer c0468a4057c94186abfa8d9f16a7869d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:10.644104 27671 consensus_queue.cc:1035] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:10.644239 27664 consensus_queue.cc:1035] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:10.647027 27676 mvcc.cc:205] Tried to move back new op lower bound from 7333194099273920512 to 7333194099244900352. Current Snapshot: MvccSnapshot[applied={T|T < 7333194099273920512}]
I20260925 10:03:10.647181 27674 mvcc.cc:205] Tried to move back new op lower bound from 7333194099273920512 to 7333194099244900352. Current Snapshot: MvccSnapshot[applied={T|T < 7333194099273920512 or (T in {7333194099273920512})}]
I20260925 10:03:10.648798 27146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:10.648887 27146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:10.649662 27146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Sending DeleteTablet for 3 replicas of tablet b99b39a2b67d4ea7975a23269766a082
I20260925 10:03:10.650020 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet b99b39a2b67d4ea7975a23269766a082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:10 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:57730
I20260925 10:03:10.650038 27283 tablet_service.cc:1514] Processing DeleteTablet for tablet b99b39a2b67d4ea7975a23269766a082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:10 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260925 10:03:10.650308 27566 tablet_service.cc:1514] Processing DeleteTablet for tablet b99b39a2b67d4ea7975a23269766a082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:10 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:46704
I20260925 10:03:10.650383 27685 tablet_replica.cc:330] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: stopping tablet replica
I20260925 10:03:10.650493 27685 raft_consensus.cc:2229] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:10.650561 27686 tablet_replica.cc:330] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: stopping tablet replica
I20260925 10:03:10.650614 27685 raft_consensus.cc:2258] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:10.650641 27686 raft_consensus.cc:2229] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:10.650554 27146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:10.650708 27146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:10.651137 27685 ts_tablet_manager.cc:1905] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:10.652292 27283 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8b954475042638706599c429f8e9b (DEFAULT_TABLE table=test-table [id=512696a4dc55418fb68d229132c68dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.652429 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8b954475042638706599c429f8e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.653215 27684 tablet_replica.cc:330] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: stopping tablet replica
I20260925 10:03:10.653324 27684 raft_consensus.cc:2229] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:10.653425 27684 raft_consensus.cc:2258] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:10.653612 27685 ts_tablet_manager.cc:1918] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:10.653687 27685 log.cc:1196] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/b99b39a2b67d4ea7975a23269766a082
I20260925 10:03:10.653753 27661 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Bootstrap starting.
I20260925 10:03:10.653818 27566 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8b954475042638706599c429f8e9b (DEFAULT_TABLE table=test-table [id=512696a4dc55418fb68d229132c68dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.653935 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8b954475042638706599c429f8e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.653975 27685 ts_tablet_manager.cc:1939] T b99b39a2b67d4ea7975a23269766a082 P 32dc1d73b4e343d7b386e1097068db2c: Deleting consensus metadata
I20260925 10:03:10.654201 27661 tablet_bootstrap.cc:654] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.654852 27661 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: No bootstrap required, opened a new log
I20260925 10:03:10.654901 27661 ts_tablet_manager.cc:1397] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.655047 27661 raft_consensus.cc:348] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.655117 27661 raft_consensus.cc:374] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.655138 27661 raft_consensus.cc:729] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.655124 27667 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Bootstrap starting.
I20260925 10:03:10.655185 27661 consensus_queue.cc:260] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.655203 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8b954475042638706599c429f8e9b (DEFAULT_TABLE table=test-table [id=512696a4dc55418fb68d229132c68dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:10.655273 27661 ts_tablet_manager.cc:1428] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.655315 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8b954475042638706599c429f8e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.655537 27667 tablet_bootstrap.cc:654] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.656107 27667 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: No bootstrap required, opened a new log
I20260925 10:03:10.656154 27667 ts_tablet_manager.cc:1397] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.656294 27667 raft_consensus.cc:348] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.656371 27667 raft_consensus.cc:374] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.656397 27667 raft_consensus.cc:729] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6b0c9de504aed9e03d5885aca4b91, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.656427 27667 consensus_queue.cc:260] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.656512 27667 ts_tablet_manager.cc:1428] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:10.656774 27660 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Bootstrap starting.
I20260925 10:03:10.657231 27660 tablet_bootstrap.cc:654] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.657279 27131 catalog_manager.cc:4841] TS 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741): tablet b99b39a2b67d4ea7975a23269766a082 (table test-table [id=97a46572c47544b5864f20d27e7f04c9]) successfully deleted
I20260925 10:03:10.657703 27684 ts_tablet_manager.cc:1905] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:10.658015 27660 tablet_bootstrap.cc:492] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: No bootstrap required, opened a new log
I20260925 10:03:10.658066 27660 ts_tablet_manager.cc:1397] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:10.658208 27660 raft_consensus.cc:348] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.658278 27660 raft_consensus.cc:374] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.658303 27660 raft_consensus.cc:729] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.658352 27660 consensus_queue.cc:260] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.658437 27660 ts_tablet_manager.cc:1428] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:03:10.651455 27686 raft_consensus.cc:2258] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:10.659332 27684 ts_tablet_manager.cc:1918] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:10.659409 27684 log.cc:1196] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/b99b39a2b67d4ea7975a23269766a082
I20260925 10:03:10.659680 27684 ts_tablet_manager.cc:1939] T b99b39a2b67d4ea7975a23269766a082 P c0468a4057c94186abfa8d9f16a7869d: Deleting consensus metadata
I20260925 10:03:10.659691 27686 ts_tablet_manager.cc:1905] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:10.660115 27130 catalog_manager.cc:4841] TS c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801): tablet b99b39a2b67d4ea7975a23269766a082 (table test-table [id=97a46572c47544b5864f20d27e7f04c9]) successfully deleted
I20260925 10:03:10.661327 27686 ts_tablet_manager.cc:1918] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:10.661386 27686 log.cc:1196] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/b99b39a2b67d4ea7975a23269766a082
I20260925 10:03:10.661659 27686 ts_tablet_manager.cc:1939] T b99b39a2b67d4ea7975a23269766a082 P e8b6b0c9de504aed9e03d5885aca4b91: Deleting consensus metadata
I20260925 10:03:10.662065 27131 catalog_manager.cc:4841] TS e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581): tablet b99b39a2b67d4ea7975a23269766a082 (table test-table [id=97a46572c47544b5864f20d27e7f04c9]) successfully deleted
W20260925 10:03:10.692094 27351 tablet.cc:2369] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:03:10.763185 27633 tablet.cc:2369] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:03:10.869144 27665 raft_consensus.cc:482] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:10.869275 27665 raft_consensus.cc:504] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.869637 27665 leader_election.cc:288] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:10.873137 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.869963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:47755 (local address 127.19.46.194:37801)
0925 10:03:10.870049 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.870052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.870145 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:03:10.870223 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.870224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.870237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.870318 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.870499 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.871175 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872148 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.872727 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872807 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.872816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.873022 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.873053 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:03:10.873157 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.869883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37801 (local address 127.19.46.193:47755)
0925 10:03:10.870027 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.870039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.870145 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.870330 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.870331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.870333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.870416 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.870418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.871255 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.871255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.872063 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.872065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.872838 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.872892 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.872897 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.873008 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.873056 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260925 10:03:10.873210 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.869960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:54615 (local address 127.19.46.195:46581)
0925 10:03:10.870019 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.870021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.870117 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:03:10.870217 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.870218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.870230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.870283 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:10.870500 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.870998 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872343 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.872872 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872930 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.872936 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.873104 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.873141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:03:10.873384 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.869883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:46581 (local address 127.19.46.193:54615)
0925 10:03:10.870032 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.870046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.870139 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.870307 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.870309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.870312 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:10.870417 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.870418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.871116 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.871117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.872249 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.872252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.872961 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.872962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.873034 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.873037 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.873088 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.873122 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":20,"thread_start_us":51,"threads_started":1}
I20260925 10:03:10.873673 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8b954475042638706599c429f8e9b" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" is_pre_election: true
I20260925 10:03:10.873790 27576 raft_consensus.cc:2454] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:10.873739 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8b954475042638706599c429f8e9b" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d" is_pre_election: true
I20260925 10:03:10.873888 27445 raft_consensus.cc:2454] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:10.873979 27235 leader_election.cc:302] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:10.874115 27665 raft_consensus.cc:2781] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:10.874167 27665 raft_consensus.cc:482] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:10.874182 27665 raft_consensus.cc:3037] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.874866 27665 raft_consensus.cc:504] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.875025 27665 leader_election.cc:288] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Requested vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:10.875142 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8b954475042638706599c429f8e9b" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d"
I20260925 10:03:10.875160 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8b954475042638706599c429f8e9b" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
I20260925 10:03:10.875214 27445 raft_consensus.cc:3037] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.875245 27576 raft_consensus.cc:3037] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:10.875914 27445 raft_consensus.cc:2454] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:10.875942 27576 raft_consensus.cc:2454] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:10.876106 27235 leader_election.cc:302] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:10.876211 27665 raft_consensus.cc:2781] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:10.876303 27665 raft_consensus.cc:686] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 1 LEADER]: Becoming Leader. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Running, Role: LEADER
I20260925 10:03:10.876349 27665 consensus_queue.cc:237] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.877032 27146 catalog_manager.cc:5495] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193). New cstate: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:03:10 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330589, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.254@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:10.909593 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.902395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.19.46.254:37713)
0925 10:03:10.902498 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:10.902500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:10.902606 (+ 106us) server_negotiation.cc:407] Connection header received
0925 10:03:10.902657 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:10.902658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:10.902668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:10.902715 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:10.903241 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.903777 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.904506 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.904696 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.904753 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.906960 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:10.906961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:10.906962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:10.906969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:10.908332 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.908730 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.908731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.908731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.908786 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:10.908985 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:10.908986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:10.908986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:10.909093 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:10.909164 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:10.909312 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:03:10.909398 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:03:10.909615 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:10.902317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37713 (local address 127.0.0.1:41762)
0925 10:03:10.902508 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:10.902519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:10.902596 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:10.902742 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:10.902743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:10.903061 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:10.903124 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.903126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.903870 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.903872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.904420 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:10.904422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:10.904817 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:10.904818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:10.904873 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:10.905425 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:10.905432 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:10.906870 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:10.908448 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.908457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.908460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.908658 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.908849 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:10.908850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:10.908850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:10.908912 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:10.909155 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:10.909163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:10.909208 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:03:10.909291 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:03:10.909501 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":73,"thread_start_us":77,"threads_started":1}
I20260925 10:03:10.911687 27146 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:41762:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:03:10.911928 27146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:10.913772 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0ec9b34b754d3daa053dcb3c3278a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42e9df6d3b244ddfa3334a3f69e15678]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:03:10.913772 27283 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0ec9b34b754d3daa053dcb3c3278a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42e9df6d3b244ddfa3334a3f69e15678]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:03:10.913965 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0ec9b34b754d3daa053dcb3c3278a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.913965 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0ec9b34b754d3daa053dcb3c3278a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.913934 27566 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0ec9b34b754d3daa053dcb3c3278a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42e9df6d3b244ddfa3334a3f69e15678]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:03:10.914079 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0ec9b34b754d3daa053dcb3c3278a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:10.915347 27667 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: Bootstrap starting.
I20260925 10:03:10.915747 27660 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: Bootstrap starting.
I20260925 10:03:10.915796 27667 tablet_bootstrap.cc:654] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.915822 27661 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: Bootstrap starting.
I20260925 10:03:10.916077 27660 tablet_bootstrap.cc:654] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.916245 27661 tablet_bootstrap.cc:654] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:10.916491 27667 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: No bootstrap required, opened a new log
I20260925 10:03:10.916545 27667 ts_tablet_manager.cc:1397] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.916698 27660 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: No bootstrap required, opened a new log
I20260925 10:03:10.916749 27660 ts_tablet_manager.cc:1397] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:10.916694 27667 raft_consensus.cc:348] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.916787 27667 raft_consensus.cc:374] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.916818 27667 raft_consensus.cc:729] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6b0c9de504aed9e03d5885aca4b91, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.916868 27667 consensus_queue.cc:260] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.916870 27660 raft_consensus.cc:348] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.916931 27660 raft_consensus.cc:374] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.916929 27661 tablet_bootstrap.cc:492] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: No bootstrap required, opened a new log
I20260925 10:03:10.916960 27660 raft_consensus.cc:729] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.916971 27661 ts_tablet_manager.cc:1397] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:10.916980 27667 ts_tablet_manager.cc:1428] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:10.916997 27660 consensus_queue.cc:260] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.917086 27660 ts_tablet_manager.cc:1428] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:10.917115 27661 raft_consensus.cc:348] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.917184 27661 raft_consensus.cc:374] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:10.917207 27661 raft_consensus.cc:729] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Initialized, Role: FOLLOWER
I20260925 10:03:10.917210 27669 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:10.917250 27661 consensus_queue.cc:260] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:10.917306 27669 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:10.917353 27661 ts_tablet_manager.cc:1428] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:10.917407 27664 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:10.917515 27664 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:10.917490 27665 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:10.917559 27665 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:11.050727 27665 raft_consensus.cc:482] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:11.050849 27665 raft_consensus.cc:504] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.051018 27665 leader_election.cc:288] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.051280 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0ec9b34b754d3daa053dcb3c3278a7" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" is_pre_election: true
I20260925 10:03:11.051292 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0ec9b34b754d3daa053dcb3c3278a7" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d" is_pre_election: true
I20260925 10:03:11.051366 27576 raft_consensus.cc:2454] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:11.051434 27445 raft_consensus.cc:2454] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:11.051546 27235 leader_election.cc:302] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:11.051661 27665 raft_consensus.cc:2781] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:11.051726 27665 raft_consensus.cc:482] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:11.051757 27665 raft_consensus.cc:3037] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.052532 27665 raft_consensus.cc:504] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.052701 27665 leader_election.cc:288] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Requested vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.052887 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0ec9b34b754d3daa053dcb3c3278a7" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
I20260925 10:03:11.052992 27576 raft_consensus.cc:3037] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.052968 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0ec9b34b754d3daa053dcb3c3278a7" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d"
I20260925 10:03:11.053072 27445 raft_consensus.cc:3037] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.053761 27445 raft_consensus.cc:2454] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:11.053776 27576 raft_consensus.cc:2454] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:11.053975 27234 leader_election.cc:302] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, c0468a4057c94186abfa8d9f16a7869d; no voters:
I20260925 10:03:11.054064 27665 raft_consensus.cc:2781] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:11.054118 27665 raft_consensus.cc:686] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [term 1 LEADER]: Becoming Leader. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Running, Role: LEADER
I20260925 10:03:11.054167 27665 consensus_queue.cc:237] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.054510 27692 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: TxnStatusTablet state changed. Reason: New leader 32dc1d73b4e343d7b386e1097068db2c. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.054636 27692 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:03:11.054816 27704 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:03:11.054898 27144 catalog_manager.cc:5495] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193). New cstate: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:11.055056 27704 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:03:11.066694 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.063639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.0.0.1:44124)
0925 10:03:11.063747 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:11.063761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:11.063823 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:11.064018 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:11.064018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:11.064019 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:03:11.064069 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.064070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.064985 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.064986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.065647 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.065649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.066128 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.066128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.066182 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.066197 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.066510 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.066522 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 10:03:11.066577 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:03:11.066630 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 10:03:11.066715 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.063742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.19.46.193:34741)
0925 10:03:11.063808 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.063810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.063822 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:11.063915 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.063916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.063938 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.064007 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.064117 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.064894 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.065727 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.066044 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.066118 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.066285 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.066420 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.066493 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.066582 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.066606 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:03:11.068150 27576 raft_consensus.cc:1261] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.068249 27445 raft_consensus.cc:1261] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.068455 27692 consensus_queue.cc:1035] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.068593 27665 consensus_queue.cc:1035] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.069324 27669 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: TxnStatusTablet state changed. Reason: New leader 32dc1d73b4e343d7b386e1097068db2c. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.069425 27669 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:11.069366 27664 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: TxnStatusTablet state changed. Reason: New leader 32dc1d73b4e343d7b386e1097068db2c. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.069505 27664 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:11.070429 27665 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.070534 27665 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:03:11.070502 27692 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.070565 27692 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P 32dc1d73b4e343d7b386e1097068db2c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:03:11.070631 27675 mvcc.cc:205] Tried to move back new op lower bound from 7333194101013016576 to 7333194100958171136. Current Snapshot: MvccSnapshot[applied={T|T < 7333194101013016576 or (T in {7333194101013016576})}]
I20260925 10:03:11.071254 27669 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.071303 27664 tablet_replica.cc:439] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } } }
I20260925 10:03:11.071365 27669 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P e8b6b0c9de504aed9e03d5885aca4b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:11.071391 27664 tablet_replica.cc:442] T 0c0ec9b34b754d3daa053dcb3c3278a7 P c0468a4057c94186abfa8d9f16a7869d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:03:11.075193 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.072669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.19.46.193:34741)
0925 10:03:11.072708 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.072711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.072720 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:11.072773 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.072774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.072784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.072828 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.072952 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.073523 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.074585 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.074768 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.074822 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.074996 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.075023 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.075063 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.075119 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.075137 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:03:11.080701 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.077896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:54821 (local address 127.19.46.193:34741)
0925 10:03:11.077938 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.077940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.078043 (+ 103us) server_negotiation.cc:407] Connection header received
0925 10:03:11.078073 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.078074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.078084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.078155 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.078263 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.079025 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.079795 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.080111 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.080182 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.080343 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.080402 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.080453 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.080569 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.080588 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260925 10:03:11.080720 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.077810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.19.46.193:54821)
0925 10:03:11.077969 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:11.077980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:11.078039 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:11.078156 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:11.078157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:11.078158 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:03:11.078212 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.078212 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.079128 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.079129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.079717 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.079719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.080189 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.080190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.080240 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.080254 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.080497 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.080499 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:03:11.080553 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:03:11.080591 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 10:03:11.083279 27576 raft_consensus.cc:1261] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.083280 27445 raft_consensus.cc:1261] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.083549 27709 consensus_queue.cc:1035] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.083659 27692 consensus_queue.cc:1035] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.103636 27144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.103766 27144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.104413 27144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Sending DeleteTablet for 3 replicas of tablet 4ae8b954475042638706599c429f8e9b
I20260925 10:03:11.104693 27283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae8b954475042638706599c429f8e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260925 10:03:11.104722 27566 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae8b954475042638706599c429f8e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:46704
I20260925 10:03:11.104751 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae8b954475042638706599c429f8e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:57730
I20260925 10:03:11.104795 27685 tablet_replica.cc:330] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: stopping tablet replica
I20260925 10:03:11.104820 27686 tablet_replica.cc:330] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: stopping tablet replica
I20260925 10:03:11.104838 27684 tablet_replica.cc:330] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: stopping tablet replica
I20260925 10:03:11.104846 27685 raft_consensus.cc:2229] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:11.104868 27686 raft_consensus.cc:2229] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:11.104887 27684 raft_consensus.cc:2229] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:11.104920 27686 raft_consensus.cc:2258] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.104921 27684 raft_consensus.cc:2258] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.104936 27685 raft_consensus.cc:2258] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.105302 27686 ts_tablet_manager.cc:1905] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.105798 27685 ts_tablet_manager.cc:1905] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.107228 27686 ts_tablet_manager.cc:1918] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:03:11.107283 27686 log.cc:1196] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/4ae8b954475042638706599c429f8e9b
I20260925 10:03:11.107327 27685 ts_tablet_manager.cc:1918] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:03:11.107367 27685 log.cc:1196] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/4ae8b954475042638706599c429f8e9b
I20260925 10:03:11.107546 27685 ts_tablet_manager.cc:1939] T 4ae8b954475042638706599c429f8e9b P 32dc1d73b4e343d7b386e1097068db2c: Deleting consensus metadata
I20260925 10:03:11.107534 27686 ts_tablet_manager.cc:1939] T 4ae8b954475042638706599c429f8e9b P e8b6b0c9de504aed9e03d5885aca4b91: Deleting consensus metadata
I20260925 10:03:11.107995 27131 catalog_manager.cc:4841] TS 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741): tablet 4ae8b954475042638706599c429f8e9b (table test-table [id=512696a4dc55418fb68d229132c68dc4]) successfully deleted
I20260925 10:03:11.108083 27131 catalog_manager.cc:4841] TS e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581): tablet 4ae8b954475042638706599c429f8e9b (table test-table [id=512696a4dc55418fb68d229132c68dc4]) successfully deleted
I20260925 10:03:11.108194 27684 ts_tablet_manager.cc:1905] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.109746 27684 ts_tablet_manager.cc:1918] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:03:11.109814 27684 log.cc:1196] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/4ae8b954475042638706599c429f8e9b
I20260925 10:03:11.110064 27684 ts_tablet_manager.cc:1939] T 4ae8b954475042638706599c429f8e9b P c0468a4057c94186abfa8d9f16a7869d: Deleting consensus metadata
I20260925 10:03:11.110733 27130 catalog_manager.cc:4841] TS c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801): tablet 4ae8b954475042638706599c429f8e9b (table test-table [id=512696a4dc55418fb68d229132c68dc4]) successfully deleted
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:03:11.134516 27195 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:11.127908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.19.46.254:37713)
0925 10:03:11.127979 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.127981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.128075 (+ 94us) server_negotiation.cc:407] Connection header received
0925 10:03:11.128133 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.128134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.128148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.128231 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:11.128871 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.129732 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.130583 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.130893 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.130968 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.134434 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41770: BlockingRecv error: recv got EOF from 127.0.0.1:41770 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM
I20260925 10:03:11.143087 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.135297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.19.46.254:37713)
0925 10:03:11.135339 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.135341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.135404 (+ 63us) server_negotiation.cc:407] Connection header received
0925 10:03:11.135483 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.135483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.135493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.135533 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:11.136143 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.136680 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.137676 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.137887 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.138010 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.140511 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:11.140513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:11.140514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:11.140522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:11.141957 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:11.142317 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:11.142318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:11.142318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:11.142377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:11.142570 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:11.142571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:11.142571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:11.142690 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:11.142791 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.142897 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.142977 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:03:11.144579 27144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:11.144767 27144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:11.146235 27283 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (DEFAULT_TABLE table=test-table [id=2d47232a92eb42cf9b592a265fdfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.146289 27566 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (DEFAULT_TABLE table=test-table [id=2d47232a92eb42cf9b592a265fdfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.146330 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (DEFAULT_TABLE table=test-table [id=2d47232a92eb42cf9b592a265fdfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.146387 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.146432 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.146421 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.147933 27667 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Bootstrap starting.
I20260925 10:03:11.147979 27660 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Bootstrap starting.
I20260925 10:03:11.148188 27661 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Bootstrap starting.
I20260925 10:03:11.148309 27660 tablet_bootstrap.cc:654] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.148310 27667 tablet_bootstrap.cc:654] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.148594 27661 tablet_bootstrap.cc:654] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.149151 27667 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: No bootstrap required, opened a new log
I20260925 10:03:11.149217 27667 ts_tablet_manager.cc:1397] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:11.149223 27661 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: No bootstrap required, opened a new log
I20260925 10:03:11.149267 27661 ts_tablet_manager.cc:1397] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:11.149371 27667 raft_consensus.cc:348] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.149459 27661 raft_consensus.cc:348] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.149535 27661 raft_consensus.cc:374] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.149537 27667 raft_consensus.cc:374] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.149555 27661 raft_consensus.cc:729] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.149572 27667 raft_consensus.cc:729] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6b0c9de504aed9e03d5885aca4b91, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.149596 27661 consensus_queue.cc:260] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.149616 27667 consensus_queue.cc:260] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.149708 27661 ts_tablet_manager.cc:1428] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:11.149715 27667 ts_tablet_manager.cc:1428] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:11.150799 27660 tablet_bootstrap.cc:492] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: No bootstrap required, opened a new log
I20260925 10:03:11.150866 27660 ts_tablet_manager.cc:1397] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:03:11.151017 27660 raft_consensus.cc:348] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.151087 27660 raft_consensus.cc:374] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.151108 27660 raft_consensus.cc:729] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.151175 27660 consensus_queue.cc:260] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.151278 27660 ts_tablet_manager.cc:1428] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:11.300643 27665 raft_consensus.cc:482] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:11.300809 27665 raft_consensus.cc:504] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.301004 27665 leader_election.cc:288] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.301216 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf4a7d2bbe14c2d83f36a7aca1b87ce" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d" is_pre_election: true
I20260925 10:03:11.301223 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf4a7d2bbe14c2d83f36a7aca1b87ce" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" is_pre_election: true
I20260925 10:03:11.301323 27445 raft_consensus.cc:2454] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:11.301322 27576 raft_consensus.cc:2454] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 0.
I20260925 10:03:11.301515 27235 leader_election.cc:302] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:11.301611 27665 raft_consensus.cc:2781] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:11.301659 27665 raft_consensus.cc:482] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:11.301681 27665 raft_consensus.cc:3037] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.302412 27665 raft_consensus.cc:504] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.302570 27665 leader_election.cc:288] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Requested vote from peers c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.302763 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf4a7d2bbe14c2d83f36a7aca1b87ce" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
I20260925 10:03:11.302839 27576 raft_consensus.cc:3037] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.303416 27576 raft_consensus.cc:2454] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:11.303553 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf4a7d2bbe14c2d83f36a7aca1b87ce" candidate_uuid: "32dc1d73b4e343d7b386e1097068db2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0468a4057c94186abfa8d9f16a7869d"
I20260925 10:03:11.303637 27445 raft_consensus.cc:3037] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.303639 27235 leader_election.cc:302] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, e8b6b0c9de504aed9e03d5885aca4b91; no voters:
I20260925 10:03:11.303735 27665 raft_consensus.cc:2781] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:11.303777 27665 raft_consensus.cc:686] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 1 LEADER]: Becoming Leader. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Running, Role: LEADER
I20260925 10:03:11.303829 27665 consensus_queue.cc:237] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.304488 27445 raft_consensus.cc:2454] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32dc1d73b4e343d7b386e1097068db2c in term 1.
I20260925 10:03:11.304425 27144 catalog_manager.cc:5495] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193). New cstate: current_term: 1 leader_uuid: "32dc1d73b4e343d7b386e1097068db2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:11.336721 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.334511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.19.46.193:34741)
0925 10:03:11.334576 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.334578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.334586 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:11.334617 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.334618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.334627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.334677 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.334808 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.335326 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.336140 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.336344 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.336416 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.336543 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.336570 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.336606 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.336644 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.336660 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:03:11.337688 27576 raft_consensus.cc:1261] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.337694 27445 raft_consensus.cc:1261] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Refusing update from remote peer 32dc1d73b4e343d7b386e1097068db2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.337939 27665 consensus_queue.cc:1035] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.338050 27692 consensus_queue.cc:1035] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.340538 27146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.340622 27146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.341185 27146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Sending DeleteTablet for 3 replicas of tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce
I20260925 10:03:11.341413 27283 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260925 10:03:11.341420 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:57730
I20260925 10:03:11.341539 27566 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:46704
I20260925 10:03:11.341545 27684 tablet_replica.cc:330] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: stopping tablet replica
I20260925 10:03:11.341614 27684 raft_consensus.cc:2229] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:11.341637 27686 tablet_replica.cc:330] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: stopping tablet replica
I20260925 10:03:11.341643 27685 tablet_replica.cc:330] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: stopping tablet replica
I20260925 10:03:11.341657 27684 raft_consensus.cc:2258] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.341686 27686 raft_consensus.cc:2229] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:11.341691 27685 raft_consensus.cc:2229] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:11.341730 27686 raft_consensus.cc:2258] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.341750 27685 raft_consensus.cc:2258] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.341885 27146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:03:11.342025 27685 ts_tablet_manager.cc:1905] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 10:03:11.342051 27146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:11.342425 27686 ts_tablet_manager.cc:1905] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.343845 27686 ts_tablet_manager.cc:1918] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:11.343915 27686 log.cc:1196] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/3cf4a7d2bbe14c2d83f36a7aca1b87ce
I20260925 10:03:11.343847 27684 ts_tablet_manager.cc:1905] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.344149 27686 ts_tablet_manager.cc:1939] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P e8b6b0c9de504aed9e03d5885aca4b91: Deleting consensus metadata
I20260925 10:03:11.344489 27131 catalog_manager.cc:4841] TS e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581): tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (table test-table [id=2d47232a92eb42cf9b592a265fdfcb55]) successfully deleted
I20260925 10:03:11.344748 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70f6a3ea5449e99376a49c367d336f (DEFAULT_TABLE table=test-table [id=7c3619413cb84a9bb612625b23dd2ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.344792 27566 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70f6a3ea5449e99376a49c367d336f (DEFAULT_TABLE table=test-table [id=7c3619413cb84a9bb612625b23dd2ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.344871 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70f6a3ea5449e99376a49c367d336f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.344870 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70f6a3ea5449e99376a49c367d336f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.345326 27684 ts_tablet_manager.cc:1918] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:11.345404 27684 log.cc:1196] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/3cf4a7d2bbe14c2d83f36a7aca1b87ce
I20260925 10:03:11.345638 27684 ts_tablet_manager.cc:1939] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P c0468a4057c94186abfa8d9f16a7869d: Deleting consensus metadata
I20260925 10:03:11.345841 27685 ts_tablet_manager.cc:1918] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:11.345906 27685 log.cc:1196] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/3cf4a7d2bbe14c2d83f36a7aca1b87ce
I20260925 10:03:11.346096 27667 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Bootstrap starting.
I20260925 10:03:11.346132 27685 ts_tablet_manager.cc:1939] T 3cf4a7d2bbe14c2d83f36a7aca1b87ce P 32dc1d73b4e343d7b386e1097068db2c: Deleting consensus metadata
I20260925 10:03:11.346200 27130 catalog_manager.cc:4841] TS c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801): tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (table test-table [id=2d47232a92eb42cf9b592a265fdfcb55]) successfully deleted
I20260925 10:03:11.346459 27131 catalog_manager.cc:4841] TS 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741): tablet 3cf4a7d2bbe14c2d83f36a7aca1b87ce (table test-table [id=2d47232a92eb42cf9b592a265fdfcb55]) successfully deleted
I20260925 10:03:11.346762 27667 tablet_bootstrap.cc:654] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.347105 27283 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70f6a3ea5449e99376a49c367d336f (DEFAULT_TABLE table=test-table [id=7c3619413cb84a9bb612625b23dd2ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:11.347223 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70f6a3ea5449e99376a49c367d336f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:11.348151 27660 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Bootstrap starting.
I20260925 10:03:11.348158 27661 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c: Bootstrap starting.
I20260925 10:03:11.348464 27660 tablet_bootstrap.cc:654] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.348500 27661 tablet_bootstrap.cc:654] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:11.348851 27667 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: No bootstrap required, opened a new log
I20260925 10:03:11.348910 27667 ts_tablet_manager.cc:1397] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:03:11.349009 27661 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c: No bootstrap required, opened a new log
I20260925 10:03:11.349061 27661 ts_tablet_manager.cc:1397] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:11.349102 27667 raft_consensus.cc:348] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.349185 27667 raft_consensus.cc:374] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.349212 27667 raft_consensus.cc:729] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6b0c9de504aed9e03d5885aca4b91, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.349181 27661 raft_consensus.cc:348] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.349269 27661 raft_consensus.cc:374] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.349261 27667 consensus_queue.cc:260] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.349298 27661 raft_consensus.cc:729] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc1d73b4e343d7b386e1097068db2c, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.349354 27667 ts_tablet_manager.cc:1428] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:11.349345 27661 consensus_queue.cc:260] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.349460 27661 ts_tablet_manager.cc:1428] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:11.349717 27660 tablet_bootstrap.cc:492] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: No bootstrap required, opened a new log
I20260925 10:03:11.349767 27660 ts_tablet_manager.cc:1397] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260925 10:03:11.349879 27660 raft_consensus.cc:348] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.349968 27660 raft_consensus.cc:374] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:11.349990 27660 raft_consensus.cc:729] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Initialized, Role: FOLLOWER
I20260925 10:03:11.350036 27660 consensus_queue.cc:260] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.350286 27660 ts_tablet_manager.cc:1428] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:11.379169 27664 raft_consensus.cc:482] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:11.379297 27664 raft_consensus.cc:504] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.379451 27664 leader_election.cc:288] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.379693 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70f6a3ea5449e99376a49c367d336f" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dc1d73b4e343d7b386e1097068db2c" is_pre_election: true
I20260925 10:03:11.379808 27303 raft_consensus.cc:2454] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 0.
I20260925 10:03:11.379743 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70f6a3ea5449e99376a49c367d336f" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" is_pre_election: true
I20260925 10:03:11.379851 27576 raft_consensus.cc:2454] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 0.
I20260925 10:03:11.380040 27376 leader_election.cc:302] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, c0468a4057c94186abfa8d9f16a7869d; no voters:
I20260925 10:03:11.380142 27664 raft_consensus.cc:2781] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:11.380183 27664 raft_consensus.cc:482] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:11.380203 27664 raft_consensus.cc:3037] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.380882 27664 raft_consensus.cc:504] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.381006 27664 leader_election.cc:288] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 election: Requested vote from peers 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741), e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581)
I20260925 10:03:11.381181 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70f6a3ea5449e99376a49c367d336f" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6b0c9de504aed9e03d5885aca4b91"
I20260925 10:03:11.381176 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70f6a3ea5449e99376a49c367d336f" candidate_uuid: "c0468a4057c94186abfa8d9f16a7869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dc1d73b4e343d7b386e1097068db2c"
I20260925 10:03:11.381250 27576 raft_consensus.cc:3037] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.381260 27303 raft_consensus.cc:3037] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:11.382033 27576 raft_consensus.cc:2454] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 1.
I20260925 10:03:11.382081 27303 raft_consensus.cc:2454] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0468a4057c94186abfa8d9f16a7869d in term 1.
I20260925 10:03:11.382217 27376 leader_election.cc:302] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32dc1d73b4e343d7b386e1097068db2c, c0468a4057c94186abfa8d9f16a7869d; no voters:
I20260925 10:03:11.382325 27664 raft_consensus.cc:2781] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:11.382486 27664 raft_consensus.cc:686] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 1 LEADER]: Becoming Leader. State: Replica: c0468a4057c94186abfa8d9f16a7869d, State: Running, Role: LEADER
I20260925 10:03:11.382555 27664 consensus_queue.cc:237] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } }
I20260925 10:03:11.383140 27144 catalog_manager.cc:5495] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0468a4057c94186abfa8d9f16a7869d (127.19.46.194). New cstate: current_term: 1 leader_uuid: "c0468a4057c94186abfa8d9f16a7869d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0468a4057c94186abfa8d9f16a7869d" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:11.389842 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.387628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.19.46.194:37801)
0925 10:03:11.387734 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.387735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.387742 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:03:11.387760 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.387761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.387771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.387823 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.387911 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.388382 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.389146 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.389349 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.389413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.389590 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.389617 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.389654 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.389701 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.389722 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260925 10:03:11.394631 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.392176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:36073 (local address 127.19.46.193:34741)
0925 10:03:11.392231 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.392233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.392352 (+ 119us) server_negotiation.cc:407] Connection header received
0925 10:03:11.392417 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.392418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.392429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.392480 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.392580 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.393045 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.393900 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.394158 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.394224 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.394405 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.394436 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.394484 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.394538 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.394555 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:03:11.394641 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.392103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.19.46.194:36073)
0925 10:03:11.392270 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:11.392281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:11.392356 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:11.392483 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:11.392484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:11.392486 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:03:11.392532 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.392534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.393124 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.393125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.393817 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.393819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.394235 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.394237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.394314 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.394328 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.394483 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.394485 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:03:11.394526 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 10:03:11.394568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 10:03:11.396771 27576 raft_consensus.cc:1261] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Refusing update from remote peer c0468a4057c94186abfa8d9f16a7869d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.396806 27303 raft_consensus.cc:1261] T 8f70f6a3ea5449e99376a49c367d336f P 32dc1d73b4e343d7b386e1097068db2c [term 1 FOLLOWER]: Refusing update from remote peer c0468a4057c94186abfa8d9f16a7869d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:11.397027 27664 consensus_queue.cc:1035] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6b0c9de504aed9e03d5885aca4b91" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.397128 27739 consensus_queue.cc:1035] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32dc1d73b4e343d7b386e1097068db2c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:11.403949 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.401597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37801 (local address 127.19.46.193:35779)
0925 10:03:11.401649 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:11.401660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:11.401703 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:11.401878 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:11.401880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:11.401881 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:03:11.401929 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.401930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.402537 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.402538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.403115 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:11.403117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.403526 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.403527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:11.403589 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.403594 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.403781 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.403783 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:03:11.403830 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:03:11.403881 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:03:11.403931 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.401637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:35779 (local address 127.19.46.194:37801)
0925 10:03:11.401680 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:11.401686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:11.401694 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:11.401779 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:11.401780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:11.401789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:11.401834 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:11.402006 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.402468 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.403185 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:11.403453 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:11.403520 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:11.403672 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:11.403708 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:11.403756 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:11.403835 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 10:03:11.403858 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:03:11.409492 27146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.409628 27146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:11.410267 27146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ca90ad2a6345e59e2b2d71fbb6bc16: Sending DeleteTablet for 3 replicas of tablet 8f70f6a3ea5449e99376a49c367d336f
I20260925 10:03:11.410511 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f70f6a3ea5449e99376a49c367d336f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:57730
I20260925 10:03:11.410616 27684 tablet_replica.cc:330] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: stopping tablet replica
I20260925 10:03:11.410620 27566 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f70f6a3ea5449e99376a49c367d336f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:11 UTC) from {username='oryx', principal='oryx/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:46704
I20260925 10:03:11.410698 27686 tablet_replica.cc:330] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: stopping tablet replica
I20260925 10:03:11.410696 27684 raft_consensus.cc:2229] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:11.410756 27684 raft_consensus.cc:2258] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.410746 27686 raft_consensus.cc:2229] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:11.410796 27686 raft_consensus.cc:2258] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:11.410946 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27218
I20260925 10:03:11.411118 27686 ts_tablet_manager.cc:1905] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.412472 27684 ts_tablet_manager.cc:1905] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:11.413591 27686 ts_tablet_manager.cc:1918] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:03:11.413652 27686 log.cc:1196] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/8f70f6a3ea5449e99376a49c367d336f
I20260925 10:03:11.413722 27684 ts_tablet_manager.cc:1918] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:03:11.413779 27684 log.cc:1196] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/8f70f6a3ea5449e99376a49c367d336f
I20260925 10:03:11.413843 27686 ts_tablet_manager.cc:1939] T 8f70f6a3ea5449e99376a49c367d336f P e8b6b0c9de504aed9e03d5885aca4b91: Deleting consensus metadata
I20260925 10:03:11.413996 27684 ts_tablet_manager.cc:1939] T 8f70f6a3ea5449e99376a49c367d336f P c0468a4057c94186abfa8d9f16a7869d: Deleting consensus metadata
I20260925 10:03:11.414134 27131 catalog_manager.cc:4841] TS e8b6b0c9de504aed9e03d5885aca4b91 (127.19.46.195:46581): tablet 8f70f6a3ea5449e99376a49c367d336f (table test-table [id=7c3619413cb84a9bb612625b23dd2ac0]) successfully deleted
I20260925 10:03:11.414390 27130 catalog_manager.cc:4841] TS c0468a4057c94186abfa8d9f16a7869d (127.19.46.194:37801): tablet 8f70f6a3ea5449e99376a49c367d336f (table test-table [id=7c3619413cb84a9bb612625b23dd2ac0]) successfully deleted
W20260925 10:03:11.418507 27131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:34741 (error 108)
I20260925 10:03:11.418759 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27359
I20260925 10:03:11.420116 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:11.419856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:34741 (local address 127.0.0.1:44152)
0925 10:03:11.419989 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:11.420053 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:34741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
W20260925 10:03:11.420286 27131 catalog_manager.cc:4584] TS 32dc1d73b4e343d7b386e1097068db2c (127.19.46.193:34741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f70f6a3ea5449e99376a49c367d336f: Network error: Client connection negotiation failed: client connection to 127.19.46.193:34741: connect: Connection refused (error 111)
I20260925 10:03:11.428211 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27500
I20260925 10:03:11.434262 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27114
2026-09-25T10:03:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2800 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27752](info): set up 2 sockets
Sep 25 10:03:11 dist-test-slave-f9rl krb5kdc[27752](info): commencing operation
krb5kdc: starting...
W20260925 10:03:12.523483 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:12 dist-test-slave-f9rl krb5kdc[27752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:03:12.644593 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44891
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:12.727914 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:12.728132 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:12.728155 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:12.729367 27765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:12.729408 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:12.729423 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:12.729454 27765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:12.729467 27765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:12.731007 27765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:12.731276 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:12.731514 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:12.736593 27773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:12.736649 27771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:12.736622 27770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:12.736756 27765 server_base.cc:1034] running on GCE node
I20260925 10:03:12.736991 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:12.737291 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:12.738458 27765 hybrid_clock.cc:648] HybridClock initialized: now 1790330592738435 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:03:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1306 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:12 dist-test-slave-f9rl krb5kdc[27778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:12 dist-test-slave-f9rl krb5kdc[27778](info): set up 2 sockets
Sep 25 10:03:12 dist-test-slave-f9rl krb5kdc[27778](info): commencing operation
krb5kdc: starting...
W20260925 10:03:13.807603 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:13 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:03:13.941887 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43603
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:14.030335 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:14.030575 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:14.030599 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:14.031741 27791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:14.031782 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:14.031796 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:14.031808 27791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:14.031821 27791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:14.033295 27791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:43603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:14.033584 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:14.033793 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:14.038857 27796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.038961 27799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.038863 27797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.039178 27791 server_base.cc:1034] running on GCE node
I20260925 10:03:14.039513 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:14.039825 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:14.041028 27791 hybrid_clock.cc:648] HybridClock initialized: now 1790330594041016 us; error 34 us; skew 500 ppm
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:14.049355 27791 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:03:14.049948 27791 webserver.cc:460] Webserver started at http://127.19.46.254:43023/ using document root <none> and password file <none>
I20260925 10:03:14.050199 27791 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:14.050267 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:14.050419 27791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:14.051540 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "66025b8b15b24884a962be999ca185ff"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "3e19ee809bf929ace8e786ab54d7d37a"
server_key_iv: "0102d4f1342e8614797ddad312df8d7b"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.051904 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "66025b8b15b24884a962be999ca185ff"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "3e19ee809bf929ace8e786ab54d7d37a"
server_key_iv: "0102d4f1342e8614797ddad312df8d7b"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.053387 27791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:14.054435 27806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.055022 27791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:14.055229 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "66025b8b15b24884a962be999ca185ff"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "3e19ee809bf929ace8e786ab54d7d37a"
server_key_iv: "0102d4f1342e8614797ddad312df8d7b"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.055356 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:14.067539 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:14.070716 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:14.070972 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:14.075253 27791 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:43603
I20260925 10:03:14.075292 27859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:43603 every 8 connection(s)
I20260925 10:03:14.075677 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:14.076610 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:14.077574 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27791
I20260925 10:03:14.077728 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:14.077951 19643 external_mini_cluster.cc:1428] Setting key 1433c4aab1d30386c2cdac817efdf950
I20260925 10:03:14.080461 27860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Bootstrap starting.
I20260925 10:03:14.081142 27860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:14.081522 27860 log.cc:824] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:14.082330 27860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: No bootstrap required, opened a new log
I20260925 10:03:14.084827 27860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.085188 27860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:14.085223 27860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66025b8b15b24884a962be999ca185ff, State: Initialized, Role: FOLLOWER
I20260925 10:03:14.085512 27860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.085754 27860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:14.085891 27860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:14.086059 27860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:14.087100 27860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.087275 27860 leader_election.cc:302] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66025b8b15b24884a962be999ca185ff; no voters:
I20260925 10:03:14.087508 27860 leader_election.cc:288] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:14.087638 27865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:14.087793 27865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 LEADER]: Becoming Leader. State: Replica: 66025b8b15b24884a962be999ca185ff, State: Running, Role: LEADER
I20260925 10:03:14.087836 27860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:14.087903 27865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.088852 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66025b8b15b24884a962be999ca185ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } } }
I20260925 10:03:14.088714 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66025b8b15b24884a962be999ca185ff. Latest consensus state: current_term: 1 leader_uuid: "66025b8b15b24884a962be999ca185ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } } }
I20260925 10:03:14.089134 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:14.089078 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:14.089591 27882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:14.090317 27882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:14.091234 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.078592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.19.46.254:43603)
0925 10:03:14.078917 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:14.078934 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:14.078998 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:03:14.079995 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:14.080009 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:14.080091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:14.080264 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:14.081144 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.082517 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.083324 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.083984 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.084117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.086873 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:14.086878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:14.086878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:14.086894 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:14.089574 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.090028 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.090030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.090030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.090115 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.090328 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.090329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.090329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.090639 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:14.090800 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:14.090925 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:03:14.091072 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260925 10:03:14.092094 27882 catalog_manager.cc:1347] Generated new cluster ID: d697e4014f46400b915b9b9c9dda66b7
I20260925 10:03:14.092152 27882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:14.131343 27882 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:14.131981 27882 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:14.137657 27882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Generated new TSK 0
I20260925 10:03:14.137887 27882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:03:14.177340 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:14.268666 27887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:14.268904 27887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:14.268931 27887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:14.270196 27887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:14.270243 27887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:14.270272 27887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:14.271754 27887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:14.272063 27887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:14.272300 27887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:14.277695 27893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.277760 27892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.277865 27887 server_base.cc:1034] running on GCE node
W20260925 10:03:14.277693 27895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.278194 27887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:14.278414 27887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:14.279554 27887 hybrid_clock.cc:648] HybridClock initialized: now 1790330594279543 us; error 28 us; skew 500 ppm
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:14.282141 27887 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:03:14.282645 27887 webserver.cc:460] Webserver started at http://127.19.46.193:36087/ using document root <none> and password file <none>
I20260925 10:03:14.282894 27887 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:14.282940 27887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:14.283077 27887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:14.284147 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "6f8f1ab6b56d46f19d37a435208a284e"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "03774b64321b3144ca5772eef9efe2fd"
server_key_iv: "2e151520714ec99f51dab395c39a2869"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.284503 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "6f8f1ab6b56d46f19d37a435208a284e"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "03774b64321b3144ca5772eef9efe2fd"
server_key_iv: "2e151520714ec99f51dab395c39a2869"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.286185 27887 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:03:14.286998 27902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.287209 27887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:14.287289 27887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "6f8f1ab6b56d46f19d37a435208a284e"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "03774b64321b3144ca5772eef9efe2fd"
server_key_iv: "2e151520714ec99f51dab395c39a2869"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.287437 27887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:14.310833 27887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:14.312963 27887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:14.313133 27887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:14.313347 27887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:14.313674 27887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:14.313714 27887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.313766 27887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:14.313791 27887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.319583 27887 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:37649
I20260925 10:03:14.319645 28016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:37649 every 8 connection(s)
I20260925 10:03:14.319965 27887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:14.323334 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 27887
I20260925 10:03:14.323426 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:14.323619 19643 external_mini_cluster.cc:1428] Setting key 295d614e18311b6ee07d58c4d3c5c8d7
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:14.330339 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.320704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:41189 (local address 127.19.46.254:43603)
0925 10:03:14.320750 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:14.320752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:14.320941 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:03:14.321506 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:14.321508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:14.321519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:14.321598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:14.322536 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.323049 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.324252 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.324656 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.324733 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.327378 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:14.327379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:14.327380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:14.327387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:14.328860 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.329306 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.329308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.329308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.329368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.329652 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.329653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.329653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.329783 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:14.329866 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:14.330151 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:03:14.330242 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:03:14.330555 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.320670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:41189)
0925 10:03:14.320824 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:14.320869 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:14.321351 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:14.321653 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:14.321656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:14.322234 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:14.322437 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.322440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.323160 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.323162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.324182 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.324185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.324766 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.324767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.324892 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.325539 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:14.325548 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:14.327254 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:14.328965 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.328974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.328976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.329215 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.329457 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.329458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.329459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.329584 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.329856 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:14.329864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:14.330011 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:03:14.330117 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:03:14.330284 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260925 10:03:14.331072 28017 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43603
I20260925 10:03:14.331166 28017 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:14.331370 28017 heartbeater.cc:511] Master 127.19.46.254:43603 requested a full tablet report, sending...
I20260925 10:03:14.331964 27823 ts_manager.cc:194] Registered new tserver with Master: 6f8f1ab6b56d46f19d37a435208a284e (127.19.46.193:37649)
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:03:14.332815 27823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:41189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:03:14.353731 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:14.436703 28022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:14.436959 28022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:14.436986 28022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:14.438318 28022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:14.438364 28022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:14.438392 28022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:14.439865 28022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:14.440124 28022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:14.440351 28022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:14.445537 28027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.445564 28028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.445537 28030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.445726 28022 server_base.cc:1034] running on GCE node
I20260925 10:03:14.446070 28022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:14.446362 28022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:14.447530 28022 hybrid_clock.cc:648] HybridClock initialized: now 1790330594447513 us; error 27 us; skew 500 ppm
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:14.450265 28022 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:03:14.450749 28022 webserver.cc:460] Webserver started at http://127.19.46.194:34579/ using document root <none> and password file <none>
I20260925 10:03:14.450986 28022 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:14.451081 28022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:14.451241 28022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:14.452346 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "57f57da7f0de4617823c066dccbf4a00"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "0acb2c5c5e0628d7f9696e565749ae06"
server_key_iv: "d9efd8f34e436fdd74acf19fb380a196"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.452709 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "57f57da7f0de4617823c066dccbf4a00"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "0acb2c5c5e0628d7f9696e565749ae06"
server_key_iv: "d9efd8f34e436fdd74acf19fb380a196"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.454196 28022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:03:14.454931 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.455098 28022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:03:14.455200 28022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "57f57da7f0de4617823c066dccbf4a00"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "0acb2c5c5e0628d7f9696e565749ae06"
server_key_iv: "d9efd8f34e436fdd74acf19fb380a196"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.455272 28022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:14.468757 28022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:14.471222 28022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:14.471395 28022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:14.471643 28022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:14.471969 28022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:14.472020 28022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.472062 28022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:14.472092 28022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.478480 28022 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:35269
I20260925 10:03:14.478528 28151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:35269 every 8 connection(s)
I20260925 10:03:14.478900 28022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:14.488870 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28022
I20260925 10:03:14.489001 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:14.489367 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.479714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57293 (local address 127.19.46.254:43603)
0925 10:03:14.479805 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:14.479808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:14.480261 (+ 453us) server_negotiation.cc:407] Connection header received
0925 10:03:14.480793 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:14.480794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:14.480805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:14.480863 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:14.481876 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.482387 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.483359 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.483557 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.483615 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.486279 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:14.486281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:14.486281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:14.486290 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:14.487765 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.488175 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.488176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.488177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.488237 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.488560 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.488561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.488561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.488681 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:14.488757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:14.489041 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 10:03:14.489129 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:03:14.489542 19643 external_mini_cluster.cc:1428] Setting key 20e10676742c02fdd343447c7d63842c
I20260925 10:03:14.489648 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.479681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:57293)
0925 10:03:14.480118 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:14.480157 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:14.480659 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:14.480953 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:14.480957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:14.481574 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:14.481775 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.481779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.482476 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.482476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.483285 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.483288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.483629 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.483630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.483730 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.484475 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:14.484485 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:14.486101 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:14.487852 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.487861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.487864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.488099 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.488339 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.488340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.488340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.488487 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.488748 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:14.488754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:14.488906 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 10:03:14.489010 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:03:14.489328 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":336,"threads_started":1}
I20260925 10:03:14.490496 28152 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43603
I20260925 10:03:14.490623 28152 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:14.490832 28152 heartbeater.cc:511] Master 127.19.46.254:43603 requested a full tablet report, sending...
I20260925 10:03:14.491361 27823 ts_manager.cc:194] Registered new tserver with Master: 57f57da7f0de4617823c066dccbf4a00 (127.19.46.194:35269)
I20260925 10:03:14.491802 27823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:57293
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:03:14.518777 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:14.605705 28157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:14.605917 28157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:14.605942 28157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:14.607523 28157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:14.607589 28157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:14.607620 28157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:14.609599 28157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:14.609915 28157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:14.610185 28157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:14.615468 28165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.615520 28157 server_base.cc:1034] running on GCE node
W20260925 10:03:14.615468 28162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.615525 28163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.615893 28157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:14.616178 28157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:14.617362 28157 hybrid_clock.cc:648] HybridClock initialized: now 1790330594617349 us; error 27 us; skew 500 ppm
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:14.620118 28157 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:03:14.621393 28157 webserver.cc:460] Webserver started at http://127.19.46.195:37135/ using document root <none> and password file <none>
I20260925 10:03:14.621691 28157 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:14.621775 28157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:14.621934 28157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:14.623103 28157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "d3002da9f0914a0e86466ff840e33c64"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "46b6b34761a182f84d1c503091ca49a9"
server_key_iv: "5063eb781a400ebe5b30e66aa11bb7cb"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.623502 28157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "d3002da9f0914a0e86466ff840e33c64"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "46b6b34761a182f84d1c503091ca49a9"
server_key_iv: "5063eb781a400ebe5b30e66aa11bb7cb"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.625039 28157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 10:03:14.625972 28172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.626183 28157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:03:14.626299 28157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "d3002da9f0914a0e86466ff840e33c64"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "46b6b34761a182f84d1c503091ca49a9"
server_key_iv: "5063eb781a400ebe5b30e66aa11bb7cb"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.626387 28157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:14.649004 28157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:14.651232 28157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:14.651401 28157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:14.651655 28157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:14.651988 28157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:14.652042 28157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.652091 28157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:14.652123 28157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.658648 28157 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:41583
I20260925 10:03:14.658699 28286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:41583 every 8 connection(s)
I20260925 10:03:14.659091 28157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:14.664651 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28157
I20260925 10:03:14.664746 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:14.664958 19643 external_mini_cluster.cc:1428] Setting key 6c9c996d4b8ba8d267367a1abbe06383
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:14.668810 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.659872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:36895 (local address 127.19.46.254:43603)
0925 10:03:14.659941 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:14.659944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:14.660152 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:03:14.660624 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:14.660625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:14.660635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:14.660690 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:14.661678 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.662254 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.663296 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.663491 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.663557 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.665918 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:14.665919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:14.665920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:14.665927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:14.667409 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.667842 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.667843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.667844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.667910 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:14.668159 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:14.668160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:14.668161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:14.668279 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:14.668348 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:14.668634 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 10:03:14.668716 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:03:14.669018 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:14.659862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:36895)
0925 10:03:14.660028 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:14.660063 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:14.660540 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:14.660740 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:14.660743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:14.661312 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:14.661593 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.661598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.662334 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.662335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.663211 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:14.663213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:14.663570 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:14.663570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:14.663674 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:14.664224 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:14.664232 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:14.665836 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:14.667524 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.667533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.667535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.667765 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.667980 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:14.667981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:14.667981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:14.668095 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:14.668351 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:14.668358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:14.668503 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:03:14.668610 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:03:14.668778 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260925 10:03:14.669528 28287 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43603
I20260925 10:03:14.669626 28287 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:14.669806 28287 heartbeater.cc:511] Master 127.19.46.254:43603 requested a full tablet report, sending...
I20260925 10:03:14.670298 27823 ts_manager.cc:194] Registered new tserver with Master: d3002da9f0914a0e86466ff840e33c64 (127.19.46.195:41583)
I20260925 10:03:14.670722 27823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:36895
I20260925 10:03:14.677376 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
Entry for principal oryx/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab.
WARNING: no policy specified for oryx/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
Entry for principal oryx/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab.
WARNING: no policy specified for oryx/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
Entry for principal oryx/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab.
I20260925 10:03:14.752470 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27887
I20260925 10:03:14.757007 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28022
I20260925 10:03:14.762033 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28157
I20260925 10:03:14.766741 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 27791
I20260925 10:03:14.772244 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43603
--webserver_interface=127.19.46.254
--webserver_port=43023
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:14.864619 28296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:14.864888 28296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:14.864913 28296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:14.866456 28296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:14.866526 28296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:14.866544 28296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:14.866557 28296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:14.866569 28296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:14.868287 28296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:43603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=43023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:14.868538 28296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:14.868731 28296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:14.874365 28302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:14.874477 28301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.874509 28296 server_base.cc:1034] running on GCE node
W20260925 10:03:14.874374 28304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:14.874869 28296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:14.875262 28296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:14.876493 28296 hybrid_clock.cc:648] HybridClock initialized: now 1790330594876470 us; error 36 us; skew 500 ppm
Sep 25 10:03:14 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330594, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:14.879249 28296 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:03:14.879689 28296 webserver.cc:460] Webserver started at http://127.19.46.254:43023/ using document root <none> and password file <none>
I20260925 10:03:14.879894 28296 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:14.879947 28296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:14.881707 28296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:03:14.882515 28311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:14.882840 28296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:14.882959 28296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "66025b8b15b24884a962be999ca185ff"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "3e19ee809bf929ace8e786ab54d7d37a"
server_key_iv: "0102d4f1342e8614797ddad312df8d7b"
server_key_version: "encryptionkey@0"
I20260925 10:03:14.883373 28296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:14.909853 28296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:14.912242 28296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:14.912364 28296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:14.916448 28296 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:43603
I20260925 10:03:14.916549 28363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:43603 every 8 connection(s)
I20260925 10:03:14.916913 28296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:14.918150 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28296
I20260925 10:03:14.918505 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:37649
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:43603
--webserver_port=36087
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:03:14.918975 28364 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:03:14.920078 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Bootstrap starting.
I20260925 10:03:14.928610 28364 log.cc:824] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:14.930794 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:03:14.931118 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Bootstrap complete.
I20260925 10:03:14.933836 28364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.934095 28364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66025b8b15b24884a962be999ca185ff, State: Initialized, Role: FOLLOWER
I20260925 10:03:14.934279 28364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.934357 28364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:14.934422 28364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:14.934473 28364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:03:14.935606 28364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.935742 28364 leader_election.cc:302] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66025b8b15b24884a962be999ca185ff; no voters:
I20260925 10:03:14.935986 28364 leader_election.cc:288] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:03:14.936031 28368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:03:14.936218 28368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [term 2 LEADER]: Becoming Leader. State: Replica: 66025b8b15b24884a962be999ca185ff, State: Running, Role: LEADER
I20260925 10:03:14.936297 28364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:14.936998 28368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } }
I20260925 10:03:14.937589 28370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66025b8b15b24884a962be999ca185ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } } }
I20260925 10:03:14.937605 28369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66025b8b15b24884a962be999ca185ff. Latest consensus state: current_term: 2 leader_uuid: "66025b8b15b24884a962be999ca185ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66025b8b15b24884a962be999ca185ff" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43603 } } }
I20260925 10:03:14.937741 28369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:14.937686 28370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:14.938000 28383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:14.938627 28383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:14.938822 28383 catalog_manager.cc:1259] Loaded cluster ID: d697e4014f46400b915b9b9c9dda66b7
I20260925 10:03:14.938905 28383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:14.940227 28383 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:14.940474 28383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66025b8b15b24884a962be999ca185ff: Loaded TSK: 0
I20260925 10:03:14.940774 28383 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:03:15.003649 28366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:15.003866 28366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:15.003892 28366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:15.005231 28366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:15.005362 28366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:15.005430 28366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:15.007016 28366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.193.keytab
--principal=oryx/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:37649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=36087
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:15.007293 28366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:15.007540 28366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:15.013839 28391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:15.013821 28393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.013866 28366 server_base.cc:1034] running on GCE node
W20260925 10:03:15.013827 28390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.014170 28366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:15.014487 28366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:15.015698 28366 hybrid_clock.cc:648] HybridClock initialized: now 1790330595015687 us; error 33 us; skew 500 ppm
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330595, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:15.018805 28366 init.cc:377] Logged in from keytab as oryx/127.19.46.193@KRBTEST.COM (short username oryx)
I20260925 10:03:15.019327 28366 webserver.cc:460] Webserver started at http://127.19.46.193:36087/ using document root <none> and password file <none>
I20260925 10:03:15.019580 28366 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:15.019665 28366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:15.021052 28366 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:03:15.021729 28400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.021917 28366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:15.022018 28366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "6f8f1ab6b56d46f19d37a435208a284e"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "03774b64321b3144ca5772eef9efe2fd"
server_key_iv: "2e151520714ec99f51dab395c39a2869"
server_key_version: "encryptionkey@0"
I20260925 10:03:15.022377 28366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:15.042434 28366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:15.044866 28366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:15.045046 28366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:15.045300 28366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:15.045697 28366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:15.045748 28366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.045794 28366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:15.045821 28366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.052707 28366 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:37649
I20260925 10:03:15.052738 28514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:37649 every 8 connection(s)
I20260925 10:03:15.053174 28366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:15.054143 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28366
I20260925 10:03:15.054473 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:35269
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:43603
--webserver_port=34579
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.066817 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.054026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:56285)
0925 10:03:15.056084 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.056144 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.056828 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.057720 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.057724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.058424 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.058645 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.058651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.060541 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.060542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.061348 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.061350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.062439 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.062441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.062676 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.063343 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.063356 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.066597 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2000,"thread_start_us":95,"threads_started":1}
W20260925 10:03:15.067308 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.054060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56285 (local address 127.19.46.254:43603)
0925 10:03:15.057058 (+ 2998us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.057068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.057112 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:03:15.057241 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.057247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.057311 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.057502 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.058791 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.060394 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.061468 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.062183 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.062307 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.067126 (+ 4819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:56285: BlockingRecv error: recv got EOF from 127.19.46.193:56285 (error 108)
Metrics: {"server-negotiator.queue_time_us":2937,"thread_start_us":86,"threads_started":1}
W20260925 10:03:15.067710 28515 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:43603 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.46.254:43603: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.073851 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.067966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:56037)
0925 10:03:15.068120 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.068130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.068176 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.068339 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.068340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.068505 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.068572 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.068573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.069605 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.069606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.070460 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.070462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.070940 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.070941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.071008 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.071507 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.071512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.073753 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
W20260925 10:03:15.074055 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.068039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56037 (local address 127.19.46.254:43603)
0925 10:03:15.068080 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.068083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.068174 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:03:15.068235 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.068236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.068249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.068321 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.068659 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.069524 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.070560 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.070850 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.070942 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.073865 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:56037: BlockingRecv error: recv got EOF from 127.19.46.193:56037 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.080088 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.074284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:58813)
0925 10:03:15.074346 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.074358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.074410 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.074631 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.074632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.074820 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.074866 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.074868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.075947 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.075948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.076860 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.076865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.077422 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.077423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.077507 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.077982 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.077988 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.080009 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:03:15.080233 28515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:03:15.080194 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.074326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:58813 (local address 127.19.46.254:43603)
0925 10:03:15.074357 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.074359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.074480 (+ 121us) server_negotiation.cc:407] Connection header received
0925 10:03:15.074512 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.074513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.074529 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.074585 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.074961 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.075828 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.076985 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.077310 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.077368 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.080100 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:58813: BlockingRecv error: recv got EOF from 127.19.46.193:58813 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260925 10:03:15.153661 28519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:15.153927 28519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:15.153954 28519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:15.155244 28519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:15.155295 28519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:15.155325 28519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:15.156857 28519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.194.keytab
--principal=oryx/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:35269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=34579
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:15.157135 28519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:15.157388 28519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:15.163022 28525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:15.163220 28526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.163374 28519 server_base.cc:1034] running on GCE node
W20260925 10:03:15.163532 28528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.163817 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:15.164079 28519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:15.165238 28519 hybrid_clock.cc:648] HybridClock initialized: now 1790330595165225 us; error 27 us; skew 500 ppm
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330595, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:15.168715 28519 init.cc:377] Logged in from keytab as oryx/127.19.46.194@KRBTEST.COM (short username oryx)
I20260925 10:03:15.169556 28519 webserver.cc:460] Webserver started at http://127.19.46.194:34579/ using document root <none> and password file <none>
I20260925 10:03:15.169836 28519 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:15.169903 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:15.171442 28519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:03:15.172399 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.172628 28519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:15.172719 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "57f57da7f0de4617823c066dccbf4a00"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "0acb2c5c5e0628d7f9696e565749ae06"
server_key_iv: "d9efd8f34e436fdd74acf19fb380a196"
server_key_version: "encryptionkey@0"
I20260925 10:03:15.173190 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:15.196192 28519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:15.198638 28519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:15.198766 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:15.199015 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:15.199450 28519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:15.199496 28519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.199532 28519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:15.199563 28519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.205817 28519 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:35269
I20260925 10:03:15.205858 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:35269 every 8 connection(s)
I20260925 10:03:15.206219 28519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:15.211346 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28519
I20260925 10:03:15.211649 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:41583
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:43603
--webserver_port=37135
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:03:15.214588 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.207050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:40141 (local address 127.19.46.254:43603)
0925 10:03:15.207135 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.207137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.207362 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:03:15.207941 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.207942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.207958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.208005 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.208975 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.209541 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.210536 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.210726 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.210806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.214530 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:40141: BlockingRecv error: recv got EOF from 127.19.46.194:40141 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:03:15.214656 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.207018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:40141)
0925 10:03:15.207236 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.207279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.207801 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.208091 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.208095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.208676 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.208883 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.208887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.209630 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.209632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.210432 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.210434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.210816 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.210817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.210952 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.211580 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.211588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.214446 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
W20260925 10:03:15.214874 28650 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:43603 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.46.254:43603: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.220310 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.215101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:43465)
0925 10:03:15.215147 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.215158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.215199 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.215318 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.215319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.215477 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.215535 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.215537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.216786 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.216788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.217307 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.217309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.217731 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.217732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.217802 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.218374 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.218378 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.220190 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 10:03:15.220322 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.215167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:43465 (local address 127.19.46.254:43603)
0925 10:03:15.215214 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.215217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.215226 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:15.215256 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.215257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.215266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.215315 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.215856 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.216682 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.217425 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.217659 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.217734 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.220275 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:43465: BlockingRecv error: recv got EOF from 127.19.46.194:43465 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.225507 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.220706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:39263)
0925 10:03:15.220761 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.220773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.220822 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.220976 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.220977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.221084 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.221116 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.221117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.222117 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.222117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.222596 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.222598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.223051 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.223052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.223101 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.223589 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.223593 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.225394 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:03:15.225586 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.220787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39263 (local address 127.19.46.254:43603)
0925 10:03:15.220850 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.220852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.220865 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:03:15.220891 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.220892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.220904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.220972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.221196 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.222032 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.222671 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.222973 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.223045 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.225520 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:39263: BlockingRecv error: recv got EOF from 127.19.46.194:39263 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 10:03:15.225696 28650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:03:15.297693 28653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:15.297912 28653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:15.297938 28653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:15.299141 28653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:15.299191 28653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:15.299220 28653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:15.300650 28653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/krb5kdc/oryx_127.19.46.195.keytab
--principal=oryx/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=37135
--tserver_master_addrs=127.19.46.254:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:15.300864 28653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:15.301055 28653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:15.306435 28661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:15.306469 28659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.306666 28653 server_base.cc:1034] running on GCE node
W20260925 10:03:15.306815 28658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:15.307060 28653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:15.307490 28653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:15.308700 28653 hybrid_clock.cc:648] HybridClock initialized: now 1790330595308681 us; error 33 us; skew 500 ppm
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330595, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:15.311538 28653 init.cc:377] Logged in from keytab as oryx/127.19.46.195@KRBTEST.COM (short username oryx)
I20260925 10:03:15.312110 28653 webserver.cc:460] Webserver started at http://127.19.46.195:37135/ using document root <none> and password file <none>
I20260925 10:03:15.312367 28653 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:15.312462 28653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:15.314080 28653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:03:15.314867 28669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.315212 28653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:15.315392 28653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "d3002da9f0914a0e86466ff840e33c64"
format_stamp: "Formatted at 2026-09-25 10:03:14 on dist-test-slave-f9rl"
server_key: "46b6b34761a182f84d1c503091ca49a9"
server_key_iv: "5063eb781a400ebe5b30e66aa11bb7cb"
server_key_version: "encryptionkey@0"
I20260925 10:03:15.315776 28653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:15.325172 28653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:15.327358 28653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:15.327538 28653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:15.327786 28653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:15.328099 28653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:15.328150 28653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.328186 28653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:15.328213 28653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:15.334580 28653 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:41583
I20260925 10:03:15.334617 28782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:41583 every 8 connection(s)
I20260925 10:03:15.334995 28653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:15.337620 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28653
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:03:15.344067 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.335948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:44915 (local address 127.19.46.254:43603)
0925 10:03:15.336036 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.336039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.336452 (+ 413us) server_negotiation.cc:407] Connection header received
0925 10:03:15.337100 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.337101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.337112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.337172 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.338484 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.339048 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.340324 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.340564 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.340633 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.343994 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:44915: BlockingRecv error: recv got EOF from 127.19.46.195:44915 (error 108)
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":43}
I20260925 10:03:15.344280 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.335944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:44915)
0925 10:03:15.336150 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.336189 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.336995 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.337229 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.337232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.338079 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.338399 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.338405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.339141 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.339142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.340234 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.340236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.340651 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.340652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.340754 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.341300 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.341311 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.343899 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
W20260925 10:03:15.344496 28783 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:43603 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.46.254:43603: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
I20260925 10:03:15.345700 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.338210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.19.46.254:43603)
0925 10:03:15.338325 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.338327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.338336 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:15.338384 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.338385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.338392 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.338462 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.339021 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.339513 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.340512 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.340772 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.340817 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.342273 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:15.342282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:15.342282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:15.342296 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:15.344059 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:15.344475 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:15.344478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:15.344479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:15.344595 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:15.344874 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:15.344875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:15.344876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:15.345123 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:15.345257 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:15.345460 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:03:15.345596 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.349255 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.344718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:56415)
0925 10:03:15.344856 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.344867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.344908 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.345020 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.345021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.345150 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.345216 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.345217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.346161 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.346162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.346822 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.346825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.347157 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.347158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.347206 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.347617 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.347622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.349206 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260925 10:03:15.349395 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.344776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:56415 (local address 127.19.46.254:43603)
0925 10:03:15.344834 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.344836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.344894 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:03:15.344963 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.344963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.344973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.345017 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.345308 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.346081 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.346918 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.347095 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.347150 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.349321 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:56415: BlockingRecv error: recv got EOF from 127.19.46.195:56415 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:15 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:15.354122 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:15.349558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:55715)
0925 10:03:15.349649 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:15.349659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:15.349713 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:15.349814 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:15.349815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:15.349949 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:15.349978 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.349980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.350804 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.350805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.351358 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:15.351360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.351738 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.351739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:15.351795 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.352243 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:15.352248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:15.354070 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 10:03:15.354249 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:15.349610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55715 (local address 127.19.46.254:43603)
0925 10:03:15.349656 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:15.349658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:15.349700 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:03:15.349752 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:15.349753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:15.349761 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:15.349803 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:15.350066 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.350666 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.351462 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:15.351647 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:15.351712 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:15.354171 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:55715: BlockingRecv error: recv got EOF from 127.19.46.195:55715 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 10:03:15.354281 28783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:16.086480 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:16.080813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:56833)
0925 10:03:16.081081 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:16.081095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:16.081182 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:16.081422 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:16.081425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:16.081687 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:16.081742 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.081745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.082517 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.082518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.083119 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.083122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.083694 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.083695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.083747 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.084228 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:16.084233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:16.086383 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":141,"threads_started":1}
W20260925 10:03:16.086632 28789 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:16.080928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56833 (local address 127.19.46.254:43603)
0925 10:03:16.081093 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:16.081094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:16.081234 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:03:16.081262 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:16.081263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:16.081275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:16.081337 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:16.081886 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.082416 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.083242 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.083600 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.083667 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.086542 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:56833: BlockingRecv error: recv got EOF from 127.19.46.193:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:16.231518 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:16.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:55725)
0925 10:03:16.226280 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:16.226294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:16.226393 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:16.226567 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:16.226569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:16.226859 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:16.226913 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.226916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.227643 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.227645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.228201 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.228203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.228581 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.228582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.228640 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.229171 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:16.229176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:16.231386 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
W20260925 10:03:16.231525 28789 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:16.226188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:55725 (local address 127.19.46.254:43603)
0925 10:03:16.226266 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:16.226269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:16.226371 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:03:16.226460 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:16.226461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:16.226476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:16.226543 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:16.227012 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.227544 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.228303 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.228494 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.228580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.231464 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:55725: BlockingRecv error: recv got EOF from 127.19.46.194:55725 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:16 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:16.359531 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:16.354706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:41949)
0925 10:03:16.354862 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:16.354878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:16.354937 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:16.355081 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:16.355082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:16.355253 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:16.355301 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.355303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.355929 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.355929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.356485 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:16.356487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.356824 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.356845 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:16.356919 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.357373 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:16.357378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:16.359453 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
W20260925 10:03:16.359604 28789 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:16.354899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:41949 (local address 127.19.46.254:43603)
0925 10:03:16.354965 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:16.354967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:16.354977 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:16.355005 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:16.355006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:16.355019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:16.355068 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:16.355370 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.355859 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.356555 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:16.356729 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:16.356778 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:16.359552 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:41949: BlockingRecv error: recv got EOF from 127.19.46.195:41949 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:17.092695 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:17.086963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:59553)
0925 10:03:17.087091 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:17.087108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:17.087157 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:17.087285 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:17.087286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:17.087469 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:17.087519 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.087521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.088444 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.088445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.088983 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.088985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.089379 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.089380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.089489 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.090137 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:17.090144 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:17.092591 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
W20260925 10:03:17.092820 28793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:17.087022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:59553 (local address 127.19.46.254:43603)
0925 10:03:17.087144 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:17.087145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:17.087154 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:17.087202 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:17.087202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:17.087212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:17.087269 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:17.087600 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.088326 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.089067 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.089286 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.089376 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.092718 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:59553: BlockingRecv error: recv got EOF from 127.19.46.193:59553 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:17.236918 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:17.232058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:36021)
0925 10:03:17.232333 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:17.232347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:17.232440 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:17.232605 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:17.232606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:17.232793 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:17.232847 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.232848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.233539 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.233540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.234087 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.234089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.234453 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.234454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.234513 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.234961 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:17.234965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:17.236858 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":89,"threads_started":1}
W20260925 10:03:17.237190 28793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:17.232150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:36021 (local address 127.19.46.254:43603)
0925 10:03:17.232344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:17.232346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:17.232422 (+ 76us) server_negotiation.cc:407] Connection header received
0925 10:03:17.232513 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:17.232514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:17.232529 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:17.232597 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:17.232923 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.233424 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.234171 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.234372 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.234442 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.237108 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:36021: BlockingRecv error: recv got EOF from 127.19.46.194:36021 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:17 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:17.365110 28795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:17.360265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:58397)
0925 10:03:17.360424 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:17.360435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:17.360507 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:17.360666 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:17.360667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:17.360840 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:17.360890 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.360892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.361671 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.361672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.362268 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:17.362270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.362631 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.362632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:17.362694 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.363209 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:17.363215 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:17.365008 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
W20260925 10:03:17.365159 28793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:17.360458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:58397 (local address 127.19.46.254:43603)
0925 10:03:17.360531 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:17.360533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:17.360545 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:17.360581 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:17.360582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:17.360595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:17.360658 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:17.360985 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.361589 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.362357 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:17.362550 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:17.362619 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:17.365092 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:58397: BlockingRecv error: recv got EOF from 127.19.46.195:58397 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:18.104979 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:18.093625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:38553)
0925 10:03:18.093759 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:18.093776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:18.093834 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:18.093957 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:18.093958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:18.094119 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:18.094168 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.094169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.094809 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.094809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.095329 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.095331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.095664 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.095664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.095703 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.102249 (+ 6546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:18.102256 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:18.104869 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260925 10:03:18.105023 28797 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:18.093666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38553 (local address 127.19.46.254:43603)
0925 10:03:18.093789 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:18.093791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:18.093822 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:03:18.093883 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:18.093883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:18.093893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:18.093946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:18.094227 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.094744 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.095410 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.095599 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.095653 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.104951 (+ 9298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:38553: BlockingRecv error: recv got EOF from 127.19.46.193:38553 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:18.242468 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:18.237387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:47249)
0925 10:03:18.237562 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:18.237578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:18.237631 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:18.237836 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:18.237838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:18.238068 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:18.238144 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.238146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.238945 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.238946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.239582 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.239585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.239964 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.239965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.240010 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.240466 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:18.240471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:18.242367 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
W20260925 10:03:18.242519 28797 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:18.237529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:47249 (local address 127.19.46.254:43603)
0925 10:03:18.237606 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:18.237609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:18.237626 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:03:18.237688 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:18.237689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:18.237703 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:18.237801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:18.238239 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.238818 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.239681 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.239881 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.239951 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.242454 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:47249: BlockingRecv error: recv got EOF from 127.19.46.194:47249 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:18 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:18.370652 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:18.365740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:51295)
0925 10:03:18.365926 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:18.365937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:18.366006 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:18.366183 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:18.366184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:18.366350 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:18.366400 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.366402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.367073 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.367074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.367613 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:18.367615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.367981 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.367982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:18.368033 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.368587 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:18.368594 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:18.370590 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
W20260925 10:03:18.370777 28797 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:18.365841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:51295 (local address 127.19.46.254:43603)
0925 10:03:18.365902 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:18.365905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:18.366002 (+ 97us) server_negotiation.cc:407] Connection header received
0925 10:03:18.366064 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:18.366065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:18.366077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:18.366139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:18.366470 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.366987 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.367693 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:18.367900 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:18.367959 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:18.370678 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:51295: BlockingRecv error: recv got EOF from 127.19.46.195:51295 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:19.118746 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:19.105702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:37465)
0925 10:03:19.105834 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:19.105846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:19.105920 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:19.106108 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:19.106109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:19.106279 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:19.106331 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.106332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.107121 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.107123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.108060 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.108068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.108618 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.108620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.108691 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.115644 (+ 6953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:19.115653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:19.118643 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
W20260925 10:03:19.119002 28802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:19.105796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:37465 (local address 127.19.46.254:43603)
0925 10:03:19.105949 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:19.105951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:19.105965 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:03:19.106008 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:19.106009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:19.106019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:19.106105 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:19.106427 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.106963 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.108149 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.108489 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.108548 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.118909 (+ 10361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:37465: BlockingRecv error: recv got EOF from 127.19.46.193:37465 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:19.247853 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:19.243037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:52269)
0925 10:03:19.243175 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:19.243187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:19.243247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:19.243417 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:19.243418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:19.243610 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:19.243659 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.243660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.244334 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.244335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.244858 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.244860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.245230 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.245231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.245287 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.245807 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:19.245812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:19.247792 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260925 10:03:19.248070 28802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:19.243170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52269 (local address 127.19.46.254:43603)
0925 10:03:19.243241 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:19.243243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:19.243252 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:19.243292 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:19.243293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:19.243304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:19.243361 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:19.243744 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.244262 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.244957 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.245144 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.245212 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.247984 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:52269: BlockingRecv error: recv got EOF from 127.19.46.194:52269 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:19 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:19.376251 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:19.371261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:57669)
0925 10:03:19.371431 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:19.371444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:19.371544 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:19.371757 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:19.371759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:19.371937 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:19.371988 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.371990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.372638 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.372638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.373146 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:19.373147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.373593 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.373594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:19.373665 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.374237 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:19.374243 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:19.376189 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
W20260925 10:03:19.376433 28802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:19.371400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:57669 (local address 127.19.46.254:43603)
0925 10:03:19.371469 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:19.371471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:19.371539 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:03:19.371622 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:19.371623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:19.371641 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:19.371722 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:19.372068 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.372568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.373220 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:19.373496 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:19.373571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:19.376336 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:57669: BlockingRecv error: recv got EOF from 127.19.46.195:57669 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:20.124413 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:20.119446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:42761)
0925 10:03:20.119617 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:20.119662 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:20.119742 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:20.119965 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:20.119966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:20.120142 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:20.120193 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.120195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.120933 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.120934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.121498 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.121499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.121923 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.121923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.121966 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.122505 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:20.122510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:20.124348 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
W20260925 10:03:20.124603 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:20.119596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:42761 (local address 127.19.46.254:43603)
0925 10:03:20.119772 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:20.119774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:20.119785 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:20.119810 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:20.119810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:20.119821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:20.119883 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:20.120297 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.120856 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.121587 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.121843 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.121915 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.124490 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:42761: BlockingRecv error: recv got EOF from 127.19.46.193:42761 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":62,"threads_started":1}
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:20.253225 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:20.248444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:42365)
0925 10:03:20.248620 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:20.248632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:20.248716 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:20.248899 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:20.248900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:20.249095 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:20.249144 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.249147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.249922 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.249923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.250442 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.250443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.250808 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.250809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.250881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.251399 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:20.251403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:20.253126 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
W20260925 10:03:20.253274 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:20.248648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:42365 (local address 127.19.46.254:43603)
0925 10:03:20.248718 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:20.248720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:20.248731 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:20.248787 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:20.248788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:20.248800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:20.248887 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:20.249239 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.249834 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.250526 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.250719 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.250806 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.253210 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:42365: BlockingRecv error: recv got EOF from 127.19.46.194:42365 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:20 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:20.381718 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:20.376792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:35199)
0925 10:03:20.377004 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:20.377013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:20.377075 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:20.377222 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:20.377223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:20.377390 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:20.377492 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.377494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.378197 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.378198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.378790 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:20.378792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.379155 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.379157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:20.379209 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.379698 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:20.379703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:20.381653 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":143,"threads_started":1}
W20260925 10:03:20.381850 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:20.376882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:35199 (local address 127.19.46.254:43603)
0925 10:03:20.376993 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:20.376995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:20.377077 (+ 82us) server_negotiation.cc:407] Connection header received
0925 10:03:20.377143 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:20.377144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:20.377158 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:20.377202 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:20.377562 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.378103 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.378885 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:20.379064 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:20.379119 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:20.381753 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:35199: BlockingRecv error: recv got EOF from 127.19.46.195:35199 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:21.130122 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:21.124973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:46989)
0925 10:03:21.125147 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:21.125160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:21.125248 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:21.125483 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:21.125484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:21.125758 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:21.125821 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.125823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.126634 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.126635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.127223 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.127225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.127645 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.127646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.127697 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.128190 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:21.128195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:21.130057 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260925 10:03:21.130303 28810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:21.125099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:46989 (local address 127.19.46.254:43603)
0925 10:03:21.125277 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:21.125279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:21.125288 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:21.125318 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:21.125324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:21.125336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:21.125456 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:21.125922 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.126551 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.127329 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.127539 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.127601 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.130194 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:46989: BlockingRecv error: recv got EOF from 127.19.46.193:46989 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:21.258797 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:21.253850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:44683)
0925 10:03:21.253976 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:21.253987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:21.254058 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:21.254255 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:21.254256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:21.254431 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:21.254485 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.254486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.255184 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.255185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.255761 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.255762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.256092 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.256093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.256134 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.256579 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:21.256584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:21.258688 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
W20260925 10:03:21.258847 28810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:21.253946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:44683 (local address 127.19.46.254:43603)
0925 10:03:21.254045 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:21.254047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:21.254059 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:21.254109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:21.254109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:21.254120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:21.254208 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:21.254590 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.255099 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.255839 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.256023 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.256076 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.258781 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:44683: BlockingRecv error: recv got EOF from 127.19.46.194:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:21 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:21.387298 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:21.382395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:44823)
0925 10:03:21.382539 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:21.382549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:21.382597 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:21.382772 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:21.382773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:21.382947 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:21.382995 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.382997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.383691 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.383691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.384243 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:21.384245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.384613 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.384615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:21.384667 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.385132 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:21.385138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:21.387224 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260925 10:03:21.387428 28810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:21.382473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:44823 (local address 127.19.46.254:43603)
0925 10:03:21.382573 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:21.382575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:21.382586 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:21.382669 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:21.382670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:21.382685 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:21.382748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:21.383090 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.383609 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.384339 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:21.384521 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:21.384580 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:21.387317 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:44823: BlockingRecv error: recv got EOF from 127.19.46.195:44823 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:22.135603 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:22.130662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:34465)
0925 10:03:22.130872 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:22.130885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:22.130953 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:22.131109 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:22.131110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:22.131274 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:22.131328 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.131329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.131992 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.131993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.132533 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.132535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.132974 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.132975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.133025 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.133576 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:22.133582 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:22.135488 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260925 10:03:22.135648 28814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:22.130766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:34465 (local address 127.19.46.254:43603)
0925 10:03:22.130901 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:22.130904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:22.130937 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:03:22.131017 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:22.131017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:22.131028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:22.131087 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:22.131391 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.131887 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.132631 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.132864 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.132944 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.135572 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:34465: BlockingRecv error: recv got EOF from 127.19.46.193:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":123,"threads_started":1}
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:22.264720 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:22.259436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:55639)
0925 10:03:22.259592 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:22.259603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:22.259668 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:22.259901 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:22.259902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:22.260136 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:22.260201 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.260203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.261077 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.261078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.261645 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.261647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.261997 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.261998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.262043 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.262493 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:22.262497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:22.264649 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260925 10:03:22.264839 28814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:22.259536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:55639 (local address 127.19.46.254:43603)
0925 10:03:22.259722 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:22.259724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:22.259743 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:03:22.259775 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:22.259776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:22.259788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:22.259861 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:22.260305 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.260987 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.261724 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.261920 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.261986 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.264754 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:55639: BlockingRecv error: recv got EOF from 127.19.46.194:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:22 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:22.393227 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:22.387937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:45161)
0925 10:03:22.388114 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:22.388135 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:22.388203 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:22.388409 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:22.388411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:22.388582 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:22.388632 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.388633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.389385 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.389386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.389952 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:22.389954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.390354 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.390355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:22.390402 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.390942 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:22.390946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:22.393129 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
W20260925 10:03:22.393288 28814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:22.388059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:45161 (local address 127.19.46.254:43603)
0925 10:03:22.388136 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:22.388137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:22.388246 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:03:22.388283 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:22.388284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:22.388298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:22.388361 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:22.388733 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.389293 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.390039 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:22.390278 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:22.390339 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:22.393210 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:45161: BlockingRecv error: recv got EOF from 127.19.46.195:45161 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:23.143499 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:23.136253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:50127)
0925 10:03:23.136465 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:23.136505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:23.136588 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:23.136975 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:23.136977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:23.137221 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:23.137304 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.137306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.138469 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.138471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.139169 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.139171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.139654 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.139656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.139736 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.140449 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:23.140454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:23.143432 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
W20260925 10:03:23.143787 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:23.136461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:50127 (local address 127.19.46.254:43603)
0925 10:03:23.136702 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:23.136704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:23.136714 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:23.136746 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:23.136747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:23.136760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:23.136840 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:23.137591 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.138339 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.139289 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.139514 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.139588 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.143688 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:50127: BlockingRecv error: recv got EOF from 127.19.46.193:50127 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:23.270464 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:23.265325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:41293)
0925 10:03:23.265496 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:23.265509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:23.265572 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:23.265728 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:23.265729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:23.265900 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:23.265949 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.265950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.266806 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.266807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.267328 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.267329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.267730 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.267731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.267805 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.268453 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:23.268457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:23.270353 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
W20260925 10:03:23.270509 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:23.265474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:41293 (local address 127.19.46.254:43603)
0925 10:03:23.265560 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:23.265562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:23.265578 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:03:23.265627 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:23.265628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:23.265639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:23.265713 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:23.266041 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.266702 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.267417 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.267594 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.267645 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.270436 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:41293: BlockingRecv error: recv got EOF from 127.19.46.194:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:23 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:23.398855 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:23.393748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:59149)
0925 10:03:23.394015 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:23.394028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:23.394126 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:23.394338 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:23.394339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:23.394535 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:23.394586 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.394587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.395273 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.395274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.395821 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:23.395823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.396186 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.396187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:23.396248 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.396770 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:23.396775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:23.398786 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
W20260925 10:03:23.398999 28818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:23.393897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:59149 (local address 127.19.46.254:43603)
0925 10:03:23.393999 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:23.394001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:23.394106 (+ 105us) server_negotiation.cc:407] Connection header received
0925 10:03:23.394186 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:23.394187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:23.394203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:23.394328 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:23.394668 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.395183 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.395908 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:23.396099 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:23.396185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:23.398915 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:59149: BlockingRecv error: recv got EOF from 127.19.46.195:59149 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:24.154120 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:24.144219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:35349)
0925 10:03:24.144400 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:24.144420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:24.144474 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:24.144856 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:24.144857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:24.145028 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:24.145088 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.145090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.145832 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.145833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.146370 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.146371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.146784 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.146786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.146843 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.151704 (+ 4861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:24.151713 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:24.154042 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":133,"threads_started":1}
W20260925 10:03:24.154301 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:24.144490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:35349 (local address 127.19.46.254:43603)
0925 10:03:24.144667 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:24.144669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:24.144682 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:03:24.144711 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:24.144712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:24.144722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:24.144789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:24.145209 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.145744 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.146470 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.146673 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.146732 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.154216 (+ 7484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:35349: BlockingRecv error: recv got EOF from 127.19.46.193:35349 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:24.276000 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:24.271056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:39579)
0925 10:03:24.271234 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:24.271248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:24.271312 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:24.271459 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:24.271460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:24.271626 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:24.271684 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.271686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.272361 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.272361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.272907 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.272909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.273252 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.273253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.273302 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.273841 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:24.273846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:24.275940 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":124,"threads_started":1}
W20260925 10:03:24.276156 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:24.271187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39579 (local address 127.19.46.254:43603)
0925 10:03:24.271256 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:24.271258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:24.271311 (+ 53us) server_negotiation.cc:407] Connection header received
0925 10:03:24.271375 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:24.271376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:24.271387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:24.271438 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:24.271764 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.272282 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.272984 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.273173 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.273224 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.276060 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:39579: BlockingRecv error: recv got EOF from 127.19.46.194:39579 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:24 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:24.405018 28824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:24.399518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:37853)
0925 10:03:24.399762 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:24.399798 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:24.399880 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:24.400066 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:24.400067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:24.400308 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:24.400361 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.400363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.401239 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.401239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.401954 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:24.401956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.402318 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.402319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:24.402361 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.402902 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:24.402907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:24.404944 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":140,"threads_started":1}
W20260925 10:03:24.405157 28822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:24.399637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:37853 (local address 127.19.46.254:43603)
0925 10:03:24.399793 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:24.399795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:24.399888 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:03:24.399937 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:24.399938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:24.399954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:24.400052 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:24.400627 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.401143 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.402040 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:24.402245 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:24.402302 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:24.405071 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:37853: BlockingRecv error: recv got EOF from 127.19.46.195:37853 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:25.159053 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:25.154631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:39099)
0925 10:03:25.154814 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:25.154824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:25.154874 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:25.155015 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:25.155016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:25.155180 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:25.155233 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.155234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.155911 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.155912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.156432 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.156434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.156749 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.156750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.156787 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.157223 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:25.157227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:25.158994 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
W20260925 10:03:25.159354 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:25.154745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:39099 (local address 127.19.46.254:43603)
0925 10:03:25.154892 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:25.154894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:25.154902 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:03:25.154924 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:25.154924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:25.154935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:25.155001 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:25.155303 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.155838 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.156503 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.156686 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.156734 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.159273 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:39099: BlockingRecv error: recv got EOF from 127.19.46.193:39099 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:25.282897 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:25.276541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:37447)
0925 10:03:25.276753 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:25.276765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:25.276834 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:25.277014 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:25.277015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:25.277186 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:25.277243 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.277245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.278159 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.278160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.278704 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.278706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.279110 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.279111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.279163 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.279724 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:25.279731 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:25.282725 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":138,"threads_started":1}
W20260925 10:03:25.282998 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:25.276662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:37447 (local address 127.19.46.254:43603)
0925 10:03:25.276718 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:25.276720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:25.276845 (+ 125us) server_negotiation.cc:407] Connection header received
0925 10:03:25.276881 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:25.276882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:25.276894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:25.276952 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:25.277423 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.278061 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.278804 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.279006 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.279065 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.282939 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:37447: BlockingRecv error: recv got EOF from 127.19.46.194:37447 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:25 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:25.410986 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:25.405600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:57661)
0925 10:03:25.405797 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:25.405811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:25.405892 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:25.406028 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:25.406028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:25.406206 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:25.406260 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.406261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.406977 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.406978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.407540 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:25.407542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.407982 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.407983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:25.408043 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.408638 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:25.408643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:25.410883 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
W20260925 10:03:25.411038 28826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:25.405777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:57661 (local address 127.19.46.254:43603)
0925 10:03:25.405831 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:25.405833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:25.405873 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:03:25.405949 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:25.405950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:25.405962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:25.406003 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:25.406383 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.406888 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.407672 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:25.407868 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:25.407934 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:25.410971 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:57661: BlockingRecv error: recv got EOF from 127.19.46.195:57661 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:26.165972 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:26.159779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:57001)
0925 10:03:26.159962 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:26.159977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:26.160053 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:26.160583 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:26.160584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:26.160765 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:26.160817 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.160819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.161709 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.161711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.162352 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.162354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.162884 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.162886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.162947 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.163604 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:26.163610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:26.165896 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
W20260925 10:03:26.166254 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:26.159924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:57001 (local address 127.19.46.254:43603)
0925 10:03:26.160183 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:26.160185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:26.160202 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:03:26.160247 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:26.160248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:26.160265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:26.160553 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:26.160917 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.161559 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.162477 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.162767 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.162836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.166199 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:57001: BlockingRecv error: recv got EOF from 127.19.46.193:57001 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:26.290279 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:26.283593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:39175)
0925 10:03:26.283868 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:26.283877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:26.283931 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:26.284080 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:26.284081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:26.284240 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:26.284294 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.284295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.285055 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.285056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.285730 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.285733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.286203 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.286204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.286252 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.287284 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:26.287294 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:26.290180 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
W20260925 10:03:26.290391 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:26.283728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39175 (local address 127.19.46.254:43603)
0925 10:03:26.283819 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:26.283821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:26.283944 (+ 123us) server_negotiation.cc:407] Connection header received
0925 10:03:26.283980 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:26.283980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:26.283993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:26.284055 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:26.284366 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.284953 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.285845 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.286071 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.286136 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.290333 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:39175: BlockingRecv error: recv got EOF from 127.19.46.194:39175 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:26 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:26.416747 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:26.411619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:53673)
0925 10:03:26.411758 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:26.411778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:26.411830 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:26.411965 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:26.411965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:26.412133 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:26.412179 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.412181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.412887 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.412888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.413505 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:26.413507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.413854 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.413855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:26.413896 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.414474 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:26.414480 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:26.416673 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260925 10:03:26.416875 28830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:26.411725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:53673 (local address 127.19.46.254:43603)
0925 10:03:26.411814 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:26.411816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:26.411831 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:03:26.411887 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:26.411888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:26.411901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:26.411943 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:26.412245 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.412797 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.413576 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:26.413757 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:26.413837 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:26.416788 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:53673: BlockingRecv error: recv got EOF from 127.19.46.195:53673 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:27.172878 28833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:27.166374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:53329)
0925 10:03:27.166546 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:27.166558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:27.166619 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:27.166937 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:27.166938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:27.167111 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:27.167159 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.167161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.168184 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.168185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.168848 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.168850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.169426 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.169427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.169574 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.170109 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:27.170114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:27.172803 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260925 10:03:27.173062 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:27.166610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:53329 (local address 127.19.46.254:43603)
0925 10:03:27.166779 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:27.166781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:27.166795 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:03:27.166819 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:27.166819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:27.166830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:27.166896 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:27.167345 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.167975 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.169020 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.169307 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.169410 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.172954 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:53329: BlockingRecv error: recv got EOF from 127.19.46.193:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:27.296180 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:27.291055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:53717)
0925 10:03:27.291297 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:27.291310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:27.291385 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:27.291545 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:27.291546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:27.291797 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:27.291844 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.291846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.292559 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.292560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.293109 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.293110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.293548 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.293549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.293608 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.294160 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:27.294164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:27.296047 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":123,"threads_started":1}
W20260925 10:03:27.296239 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:27.291185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:53717 (local address 127.19.46.254:43603)
0925 10:03:27.291290 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:27.291292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:27.291368 (+ 76us) server_negotiation.cc:407] Connection header received
0925 10:03:27.291449 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:27.291450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:27.291471 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:27.291540 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:27.291942 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.292470 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.293202 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.293400 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.293547 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.296165 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:53717: BlockingRecv error: recv got EOF from 127.19.46.194:53717 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:27 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:27.422698 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:27.417457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:54537)
0925 10:03:27.417656 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:27.417678 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:27.417751 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:27.417899 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:27.417900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:27.418076 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:27.418123 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.418125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.418905 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.418906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.419494 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:27.419496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.419912 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.419913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:27.419958 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.420437 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:27.420441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:27.422596 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
W20260925 10:03:27.422747 28834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:27.417608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:54537 (local address 127.19.46.254:43603)
0925 10:03:27.417679 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:27.417681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:27.417745 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:03:27.417818 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:27.417819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:27.417832 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:27.417875 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:27.418223 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.418828 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.419594 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:27.419823 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:27.419893 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:27.422680 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:54537: BlockingRecv error: recv got EOF from 127.19.46.195:54537 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:28.178608 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:28.173486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:57823)
0925 10:03:28.173652 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:28.173666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:28.173738 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:28.173925 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:28.173926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:28.174100 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:28.174162 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.174164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.174961 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.174961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.175566 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.175568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.175930 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.175931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.175979 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.176583 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:28.176588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:28.178523 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
W20260925 10:03:28.178711 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:28.173531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:57823 (local address 127.19.46.254:43603)
0925 10:03:28.173714 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:28.173716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:28.173726 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:28.173807 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:28.173807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:28.173818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:28.173883 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:28.174270 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.174885 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.175631 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.175846 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.175902 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.178642 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:57823: BlockingRecv error: recv got EOF from 127.19.46.193:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:28.302201 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:28.296975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:57803)
0925 10:03:28.297291 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:28.297308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:28.297389 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:28.297655 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:28.297656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:28.297843 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:28.297895 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.297896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.298620 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.298621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.299184 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.299186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.299592 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.299593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.299644 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.300202 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:28.300207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:28.302100 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":186,"threads_started":1}
W20260925 10:03:28.302588 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:28.297269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57803 (local address 127.19.46.254:43603)
0925 10:03:28.297361 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:28.297363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:28.297374 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:03:28.297539 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:28.297540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:28.297552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:28.297615 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:28.297990 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.298513 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.299300 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.299495 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.299561 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.302512 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:57803: BlockingRecv error: recv got EOF from 127.19.46.194:57803 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:28 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:28.429171 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:28.423288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:32895)
0925 10:03:28.423580 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:28.423594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:28.423682 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:28.423881 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:28.423882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:28.424091 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:28.424140 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.424142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.425119 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.425121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.425923 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:28.425926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.426449 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.426450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:28.426501 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.427076 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:28.427081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:28.429094 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
W20260925 10:03:28.429296 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:28.423389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:32895 (local address 127.19.46.254:43603)
0925 10:03:28.423569 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:28.423571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:28.423658 (+ 87us) server_negotiation.cc:407] Connection header received
0925 10:03:28.423774 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:28.423775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:28.423799 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:28.423876 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:28.424251 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.424990 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.426017 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:28.426211 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:28.426276 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:28.429211 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:32895: BlockingRecv error: recv got EOF from 127.19.46.195:32895 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:29.194854 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:29.179187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:60759)
0925 10:03:29.179349 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:29.179362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:29.179436 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:29.179645 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:29.179646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:29.179827 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:29.179876 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.179877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.180615 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.180616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.181155 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.181157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.181570 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.181571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.181619 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.192647 (+ 11028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:29.192654 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:29.194777 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260925 10:03:29.195047 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:29.179307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:60759 (local address 127.19.46.254:43603)
0925 10:03:29.179488 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:29.179490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:29.179499 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:29.179524 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:29.179524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:29.179535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:29.179602 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:29.179979 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.180509 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.181257 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.181482 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.181545 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.194926 (+ 13381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:60759: BlockingRecv error: recv got EOF from 127.19.46.193:60759 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:29.307857 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:29.302871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:60049)
0925 10:03:29.303009 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:29.303024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:29.303113 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:29.303303 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:29.303304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:29.303494 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:29.303541 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.303543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.304226 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.304227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.304777 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.304779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.305145 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.305146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.305206 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.305763 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:29.305767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:29.307774 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260925 10:03:29.307919 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:29.303010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:60049 (local address 127.19.46.254:43603)
0925 10:03:29.303087 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:29.303089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:29.303101 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:29.303218 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:29.303220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:29.303234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:29.303287 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:29.303630 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.304137 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.304866 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.305065 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.305143 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.307870 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:60049: BlockingRecv error: recv got EOF from 127.19.46.194:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:29 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.195@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:29.435787 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:29.429761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.195:49829)
0925 10:03:29.430033 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:29.430057 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:29.430145 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:29.430379 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:29.430380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:29.430783 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:29.430840 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.430845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.431568 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.431575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.432212 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:29.432214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.432856 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.432857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:29.432910 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.433412 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:29.433417 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:29.435678 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":73,"threads_started":1}
W20260925 10:03:29.435863 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:29.429857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:49829 (local address 127.19.46.254:43603)
0925 10:03:29.429906 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:29.429909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:29.430129 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:03:29.430261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:29.430262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:29.430280 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:29.430353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:29.430967 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.431468 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.432560 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:29.432759 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:29.432830 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:29.435786 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.195:49829: BlockingRecv error: recv got EOF from 127.19.46.195:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 10:03:30 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:30 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.193@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:30.201807 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.195689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.193:47829)
0925 10:03:30.195838 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:30.195853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:30.195905 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:30.196155 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:30.196156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:30.196324 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:30.196378 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.196379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.197152 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.197153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.197870 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.197873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.198383 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.198384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.198429 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.198948 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:30.198954 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:30.201743 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260925 10:03:30.202005 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:30.195793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:47829 (local address 127.19.46.254:43603)
0925 10:03:30.195956 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.195959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.195971 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:03:30.196011 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.196012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.196024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.196102 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.196470 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.197052 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.197998 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.198316 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.198374 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.201915 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.193:47829: BlockingRecv error: recv got EOF from 127.19.46.193:47829 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
Sep 25 10:03:30 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:03:30 dist-test-slave-f9rl krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.46.194@KRBTEST.COM for oryx/127.19.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:03:30.313282 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.308555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43603 (local address 127.19.46.194:37241)
0925 10:03:30.308740 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:30.308754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:30.308828 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:30.309104 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:30.309110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:30.309337 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:30.309386 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.309388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.310119 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.310119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.310651 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.310653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.310970 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.310971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.311011 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.311521 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:30.311526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:30.313215 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.46.254:43603: Server oryx/127.19.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
W20260925 10:03:30.313508 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:30.308675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:37241 (local address 127.19.46.254:43603)
0925 10:03:30.308733 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.308735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.308830 (+ 95us) server_negotiation.cc:407] Connection header received
0925 10:03:30.308921 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.308922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.308936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.309001 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.309504 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.310032 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.310724 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.310907 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.310957 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.313407 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.46.194:37241: BlockingRecv error: recv got EOF from 127.19.46.194:37241 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:03:30.341589 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28366
I20260925 10:03:30.346191 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28519
I20260925 10:03:30.350901 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28653
I20260925 10:03:30.355285 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28296
2026-09-25T10:03:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17612 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:30Z Disabled control of system clock
I20260925 10:03:30.383591 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43677
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:03:30.465674 28851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:30.465895 28851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:30.465916 28851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:30.467173 28851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:30.467221 28851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:30.467234 28851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:30.467294 28851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:30.467329 28851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:30.468706 28851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:43677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:30.468986 28851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:30.469296 28851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:30.475409 28856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.475409 28851 server_base.cc:1034] running on GCE node
W20260925 10:03:30.475504 28859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:30.475471 28857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.475989 28851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:30.476326 28851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:30.477521 28851 hybrid_clock.cc:648] HybridClock initialized: now 1790330610477513 us; error 47 us; skew 500 ppm
I20260925 10:03:30.478906 28851 webserver.cc:460] Webserver started at http://127.19.46.254:35581/ using document root <none> and password file <none>
I20260925 10:03:30.479208 28851 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:30.479300 28851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:30.479524 28851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:30.480675 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "9536f914128f4165a28b95e22806fe20"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "18b292f513efac1e5bef71a1d60b7bb9"
server_key_iv: "9e26ed4eb7a9273d1ad14268d042fd0c"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.481069 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "9536f914128f4165a28b95e22806fe20"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "18b292f513efac1e5bef71a1d60b7bb9"
server_key_iv: "9e26ed4eb7a9273d1ad14268d042fd0c"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.482570 28851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:30.483769 28865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.484108 28851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:30.484292 28851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "9536f914128f4165a28b95e22806fe20"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "18b292f513efac1e5bef71a1d60b7bb9"
server_key_iv: "9e26ed4eb7a9273d1ad14268d042fd0c"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.484611 28851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:30.497390 28851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:30.497833 28851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:30.498013 28851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:30.502715 28851 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:43677
I20260925 10:03:30.502837 28917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:43677 every 8 connection(s)
I20260925 10:03:30.503178 28851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:30.503949 28918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:30.506774 28918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20: Bootstrap starting.
I20260925 10:03:30.507361 28918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:30.507617 28918 log.cc:824] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:30.508383 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28851
I20260925 10:03:30.508421 28918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20: No bootstrap required, opened a new log
I20260925 10:03:30.508479 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:30.508683 19643 external_mini_cluster.cc:1428] Setting key 3298b8df39c5863471c55b8bfc215193
I20260925 10:03:30.510476 28918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } }
I20260925 10:03:30.510628 28918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:30.510651 28918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9536f914128f4165a28b95e22806fe20, State: Initialized, Role: FOLLOWER
I20260925 10:03:30.510810 28918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } }
I20260925 10:03:30.510901 28918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:30.510936 28918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:30.510986 28918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:30.511587 28918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } }
I20260925 10:03:30.511715 28918 leader_election.cc:302] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9536f914128f4165a28b95e22806fe20; no voters:
I20260925 10:03:30.511888 28918 leader_election.cc:288] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:30.511973 28923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:30.512197 28918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:30.512212 28923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [term 1 LEADER]: Becoming Leader. State: Replica: 9536f914128f4165a28b95e22806fe20, State: Running, Role: LEADER
I20260925 10:03:30.512409 28923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } }
I20260925 10:03:30.512794 28924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9536f914128f4165a28b95e22806fe20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } } }
I20260925 10:03:30.512893 28924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:30.512949 28925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9536f914128f4165a28b95e22806fe20. Latest consensus state: current_term: 1 leader_uuid: "9536f914128f4165a28b95e22806fe20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9536f914128f4165a28b95e22806fe20" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43677 } } }
I20260925 10:03:30.513015 28925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20 [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:30.513232 28932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:30.514322 28932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:30.516263 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.509292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.19.46.254:43677)
0925 10:03:30.509544 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.509552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.509583 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:03:30.510480 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.510489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.510576 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.510786 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.511033 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.512717 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.513970 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.514609 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.514736 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.515505 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:30.515509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:30.515510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:30.515746 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:30.515781 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:30.515793 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:30.515873 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:30.515951 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:03:30.516035 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260925 10:03:30.516618 28932 catalog_manager.cc:1347] Generated new cluster ID: 4c81faa38c6744f8a07a27ce4e2fddcf
I20260925 10:03:30.516675 28932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:30.549518 28932 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:30.550757 28932 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:30.556353 28932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9536f914128f4165a28b95e22806fe20: Generated new TSK 0
I20260925 10:03:30.556607 28932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:30.560446 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--builtin_ntp_servers=127.19.46.212:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:03:30.646348 28942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:30.646553 28942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:30.646574 28942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:30.647778 28942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:30.647821 28942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:30.647847 28942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:30.649343 28942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:30.649614 28942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:30.649938 28942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:30.655431 28947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:30.655452 28948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.655789 28942 server_base.cc:1034] running on GCE node
W20260925 10:03:30.655413 28950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.656129 28942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:30.656347 28942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:30.657510 28942 hybrid_clock.cc:648] HybridClock initialized: now 1790330610657495 us; error 40 us; skew 500 ppm
I20260925 10:03:30.658839 28942 webserver.cc:460] Webserver started at http://127.19.46.193:38993/ using document root <none> and password file <none>
I20260925 10:03:30.659149 28942 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:30.659225 28942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:30.659379 28942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:30.660501 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "a8cec90d4073443d81aa98db94d3e9b0"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "2b7a81a67e5665ba91ced29dc08a6ac6"
server_key_iv: "9d33fa2508b37c1af21e5b77f38d1207"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.660871 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "a8cec90d4073443d81aa98db94d3e9b0"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "2b7a81a67e5665ba91ced29dc08a6ac6"
server_key_iv: "9d33fa2508b37c1af21e5b77f38d1207"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.662420 28942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:30.663273 28956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.663523 28942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:30.663646 28942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "a8cec90d4073443d81aa98db94d3e9b0"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "2b7a81a67e5665ba91ced29dc08a6ac6"
server_key_iv: "9d33fa2508b37c1af21e5b77f38d1207"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.663766 28942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:30.686815 28942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:30.687198 28942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:30.687410 28942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:30.687682 28942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:30.688090 28942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:30.688165 28942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.688212 28942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:30.688244 28942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.694787 28942 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:44325
I20260925 10:03:30.694895 29071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:44325 every 8 connection(s)
I20260925 10:03:30.695271 28942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:30.695963 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 28942
I20260925 10:03:30.696076 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:30.696326 19643 external_mini_cluster.cc:1428] Setting key 0150ab8c547c4f90bbe4f8b7eaa040ec
I20260925 10:03:30.697392 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--builtin_ntp_servers=127.19.46.212:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:03:30.702440 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.696166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:50573 (local address 127.19.46.254:43677)
0925 10:03:30.696232 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.696235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.696469 (+ 234us) server_negotiation.cc:407] Connection header received
0925 10:03:30.697000 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.697001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.697014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.697065 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.697964 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.698856 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.700606 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.700928 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.701014 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.702116 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:30.702118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:30.702118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:30.702127 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:30.702143 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:30.702147 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:30.702227 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:30.702310 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:03:30.702346 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:03:30.702697 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.696139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43677 (local address 127.19.46.193:50573)
0925 10:03:30.696339 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:30.696382 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:30.696905 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:30.697154 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:30.697159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:30.697170 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:30.697866 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.697873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.699005 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.699006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.700448 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.700464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.701059 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.701060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.701263 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.701940 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:30.701940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:30.701972 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:30.701999 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:30.702227 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:30.702234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:30.702237 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:30.702316 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:03:30.702434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260925 10:03:30.703426 29072 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43677
I20260925 10:03:30.703526 29072 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:30.703768 29072 heartbeater.cc:511] Master 127.19.46.254:43677 requested a full tablet report, sending...
I20260925 10:03:30.704355 28882 ts_manager.cc:194] Registered new tserver with Master: a8cec90d4073443d81aa98db94d3e9b0 (127.19.46.193:44325)
I20260925 10:03:30.704957 28882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.193:50573
W20260925 10:03:30.788015 29075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:30.788298 29075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:30.788339 29075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:30.790161 29075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:30.790242 29075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:30.790292 29075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:30.791865 29075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:30.792171 29075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:30.792536 29075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:30.798044 29080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:30.798045 29081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.798316 29075 server_base.cc:1034] running on GCE node
W20260925 10:03:30.798092 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.798904 29075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:30.799235 29075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:30.800408 29075 hybrid_clock.cc:648] HybridClock initialized: now 1790330610800380 us; error 54 us; skew 500 ppm
I20260925 10:03:30.801967 29075 webserver.cc:460] Webserver started at http://127.19.46.194:42269/ using document root <none> and password file <none>
I20260925 10:03:30.802245 29075 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:30.802322 29075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:30.802490 29075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:30.803682 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "12caee8e33ea45f2bc5c6aaa813623b9"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "9d83378420f47573eb0db8834807c924"
server_key_iv: "a090f6d5f41337fadf0b4504def45380"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.804082 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "12caee8e33ea45f2bc5c6aaa813623b9"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "9d83378420f47573eb0db8834807c924"
server_key_iv: "a090f6d5f41337fadf0b4504def45380"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.805613 29075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:30.806419 29089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.806653 29075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:30.806762 29075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "12caee8e33ea45f2bc5c6aaa813623b9"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "9d83378420f47573eb0db8834807c924"
server_key_iv: "a090f6d5f41337fadf0b4504def45380"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.806854 29075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:30.815899 29075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:30.816238 29075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:30.816392 29075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:30.816636 29075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:30.816989 29075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:30.817046 29075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.817092 29075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:30.817126 29075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.823071 29075 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34877
I20260925 10:03:30.823153 29203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34877 every 8 connection(s)
I20260925 10:03:30.823499 29075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:30.824162 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29075
I20260925 10:03:30.824265 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:30.824473 19643 external_mini_cluster.cc:1428] Setting key b7a91dae0ade5f59c12792a9622de30e
I20260925 10:03:30.825659 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--builtin_ntp_servers=127.19.46.212:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:03:30.829934 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.824376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:59125 (local address 127.19.46.254:43677)
0925 10:03:30.824449 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.824452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.824699 (+ 247us) server_negotiation.cc:407] Connection header received
0925 10:03:30.825413 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.825415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.825430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.825546 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.826306 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.827098 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.828192 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.828406 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.828503 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.829602 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:30.829604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:30.829604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:30.829609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:30.829625 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:30.829630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:30.829708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:30.829820 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 10:03:30.829850 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:03:30.830157 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.824376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43677 (local address 127.19.46.194:59125)
0925 10:03:30.824572 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:30.824616 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:30.825290 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:30.825613 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:30.825617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:30.825626 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:30.826209 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.826214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.827219 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.827220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.828072 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.828075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.828515 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.828516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.828668 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.829377 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:30.829377 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:30.829407 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:30.829431 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:30.829711 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:30.829718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:30.829721 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:30.829811 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:03:30.829918 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 10:03:30.830708 29204 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43677
I20260925 10:03:30.830832 29204 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:30.831077 29204 heartbeater.cc:511] Master 127.19.46.254:43677 requested a full tablet report, sending...
I20260925 10:03:30.831534 28882 ts_manager.cc:194] Registered new tserver with Master: 12caee8e33ea45f2bc5c6aaa813623b9 (127.19.46.194:34877)
I20260925 10:03:30.832183 28882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.194:59125
W20260925 10:03:30.912559 29207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:30.912815 29207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:30.912842 29207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:30.914235 29207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:30.914286 29207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:30.914314 29207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:30.915640 29207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:30.915853 29207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:30.916121 29207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:30.921272 29215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:30.921605 29213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:30.921828 29212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:30.922320 29207 server_base.cc:1034] running on GCE node
I20260925 10:03:30.922539 29207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:30.922850 29207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:30.924031 29207 hybrid_clock.cc:648] HybridClock initialized: now 1790330610924002 us; error 43 us; skew 500 ppm
I20260925 10:03:30.925743 29207 webserver.cc:460] Webserver started at http://127.19.46.195:37793/ using document root <none> and password file <none>
I20260925 10:03:30.926056 29207 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:30.926128 29207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:30.926321 29207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:30.927802 29207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "918effe07c5f4abbb67003df1372335d"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "dcf616730d366c0f611db9c4bb84e752"
server_key_iv: "43c22ca9ca7ec0c9767decbb33ddc857"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.928280 29207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "918effe07c5f4abbb67003df1372335d"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "dcf616730d366c0f611db9c4bb84e752"
server_key_iv: "43c22ca9ca7ec0c9767decbb33ddc857"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.930025 29207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:30.930958 29221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.931141 29207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:30.931238 29207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "918effe07c5f4abbb67003df1372335d"
format_stamp: "Formatted at 2026-09-25 10:03:30 on dist-test-slave-f9rl"
server_key: "dcf616730d366c0f611db9c4bb84e752"
server_key_iv: "43c22ca9ca7ec0c9767decbb33ddc857"
server_key_version: "encryptionkey@0"
I20260925 10:03:30.931316 29207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:30.954039 29207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:30.954730 29207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:30.954916 29207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:30.955193 29207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:30.955552 29207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:30.955610 29207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.955653 29207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:30.955682 29207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:30.961793 29207 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:33821
I20260925 10:03:30.961838 29335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:33821 every 8 connection(s)
I20260925 10:03:30.962213 29207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:30.967711 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.963017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55889 (local address 127.19.46.254:43677)
0925 10:03:30.963083 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.963085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.963325 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:03:30.963852 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.963853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.963864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.963924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.964619 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.965146 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.966288 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.966493 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.966571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.967438 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:30.967439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:30.967440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:30.967444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:30.967459 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:30.967463 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:30.967519 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:30.967617 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:03:30.967644 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:03:30.967924 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:30.963001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:43677 (local address 127.19.46.195:55889)
0925 10:03:30.963188 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:30.963229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:30.963728 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:30.964000 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:30.964004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:30.964012 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:30.964517 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.964521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.965255 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.965256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.966161 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:30.966164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:30.966586 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:30.966587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:30.966700 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:30.967229 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:30.967230 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:30.967259 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:30.967294 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:30.967539 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:30.967543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:30.967545 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:03:30.967601 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:03:30.967686 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 10:03:30.968545 29336 heartbeater.cc:347] Connected to a master server at 127.19.46.254:43677
I20260925 10:03:30.968703 29336 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:30.968954 29336 heartbeater.cc:511] Master 127.19.46.254:43677 requested a full tablet report, sending...
I20260925 10:03:30.969379 28882 ts_manager.cc:194] Registered new tserver with Master: 918effe07c5f4abbb67003df1372335d (127.19.46.195:33821)
I20260925 10:03:30.969774 28882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.195:55889
I20260925 10:03:30.971654 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29207
I20260925 10:03:30.971750 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:30.971959 19643 external_mini_cluster.cc:1428] Setting key f6dc3c59271c46254b3793ee91aecd78
I20260925 10:03:30.972910 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:03:30.974852 28922 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:30.974338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.19.46.254:43677)
0925 10:03:30.974411 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:30.974413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:30.974504 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:03:30.974537 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:30.974537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:30.974548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:30.974590 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:30.974773 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48726: BlockingRecv error: recv got EOF from 127.0.0.1:48726 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:03:30.974869 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28942
I20260925 10:03:30.980427 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29075
I20260925 10:03:30.984850 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29207
I20260925 10:03:30.989236 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 28851
2026-09-25T10:03:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (632 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:31Z Disabled control of system clock
I20260925 10:03:31.016525 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38785
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:03:31.103181 29347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:31.103415 29347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:31.103438 29347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:31.104602 29347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:31.104642 29347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:31.104655 29347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:31.104668 29347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:31.104681 29347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:31.106151 29347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:31.106384 29347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:31.106652 29347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:31.111739 29353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.111716 29352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.111702 29355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:31.111937 29347 server_base.cc:1034] running on GCE node
I20260925 10:03:31.112257 29347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:31.112510 29347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:31.113692 29347 hybrid_clock.cc:648] HybridClock initialized: now 1790330611113682 us; error 36 us; skew 500 ppm
I20260925 10:03:31.115088 29347 webserver.cc:460] Webserver started at http://127.19.46.254:44543/ using document root <none> and password file <none>
I20260925 10:03:31.115350 29347 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:31.115422 29347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:31.115579 29347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:31.116699 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "a903f7c2218c4a6995354ed0d9166b4a"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "a377d1f6f4fa43c48a5211268b85ee0f"
server_key_iv: "6ac04ecc5a20eef45e7756b1df74f3fe"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.117071 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "a903f7c2218c4a6995354ed0d9166b4a"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "a377d1f6f4fa43c48a5211268b85ee0f"
server_key_iv: "6ac04ecc5a20eef45e7756b1df74f3fe"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.118531 29347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:31.119761 29361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.120132 29347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.120285 29347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "a903f7c2218c4a6995354ed0d9166b4a"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "a377d1f6f4fa43c48a5211268b85ee0f"
server_key_iv: "6ac04ecc5a20eef45e7756b1df74f3fe"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.120391 29347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:31.123965 29347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:31.124255 29347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:31.124398 29347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:31.128371 29347 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:38785
I20260925 10:03:31.128407 29413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:38785 every 8 connection(s)
I20260925 10:03:31.128783 29347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:31.129572 29414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:31.131148 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29347
I20260925 10:03:31.131294 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:31.131479 19643 external_mini_cluster.cc:1428] Setting key 895dfbdcded069eea0783b0ca1afc425
I20260925 10:03:31.132812 29414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a: Bootstrap starting.
I20260925 10:03:31.133585 29414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:31.133970 29414 log.cc:824] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:31.134217 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.132072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.19.46.254:38785)
0925 10:03:31.132270 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:31.132277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:31.132302 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:03:31.132715 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:31.132721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:31.132768 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:31.132912 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:31.133661 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:31.133666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:31.133666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:31.133671 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:31.133729 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:31.133739 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:31.133850 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:31.133928 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:03:31.134001 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260925 10:03:31.134740 29414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a: No bootstrap required, opened a new log
I20260925 10:03:31.136098 29414 raft_consensus.cc:348] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } }
I20260925 10:03:31.136245 29414 raft_consensus.cc:374] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:31.136268 29414 raft_consensus.cc:729] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a903f7c2218c4a6995354ed0d9166b4a, State: Initialized, Role: FOLLOWER
I20260925 10:03:31.136397 29414 consensus_queue.cc:260] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } }
I20260925 10:03:31.136480 29414 raft_consensus.cc:388] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:31.136523 29414 raft_consensus.cc:482] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:31.136574 29414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:31.137414 29414 raft_consensus.cc:504] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } }
I20260925 10:03:31.137590 29414 leader_election.cc:302] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a903f7c2218c4a6995354ed0d9166b4a; no voters:
I20260925 10:03:31.137780 29414 leader_election.cc:288] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:31.137816 29419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:31.138005 29419 raft_consensus.cc:686] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [term 1 LEADER]: Becoming Leader. State: Replica: a903f7c2218c4a6995354ed0d9166b4a, State: Running, Role: LEADER
I20260925 10:03:31.138046 29414 sys_catalog.cc:564] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:31.138145 29419 consensus_queue.cc:237] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } }
I20260925 10:03:31.138553 29420 sys_catalog.cc:455] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a903f7c2218c4a6995354ed0d9166b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } } }
I20260925 10:03:31.138564 29421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a903f7c2218c4a6995354ed0d9166b4a. Latest consensus state: current_term: 1 leader_uuid: "a903f7c2218c4a6995354ed0d9166b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a903f7c2218c4a6995354ed0d9166b4a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38785 } } }
I20260925 10:03:31.138702 29420 sys_catalog.cc:458] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:31.138713 29421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:31.139002 29428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:31.139549 29428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:31.141613 29428 catalog_manager.cc:1347] Generated new cluster ID: 5290c0be87ba418cb14f0cdd295b142c
I20260925 10:03:31.141752 29428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:03:31.150810 29428 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:31.151010 29428 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:31.156914 29428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a903f7c2218c4a6995354ed0d9166b4a: Generated new TSK 0
I20260925 10:03:31.157181 29428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:03:31.160274 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--builtin_ntp_servers=127.19.46.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:03:31.247352 29438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:31.247562 29438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:31.247586 29438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:31.248757 29438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:31.248799 29438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:31.248826 29438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:31.250238 29438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:31.250494 29438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:31.250851 29438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:31.255954 29446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.255975 29443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.256000 29444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:31.256237 29438 server_base.cc:1034] running on GCE node
I20260925 10:03:31.256616 29438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:31.256877 29438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:31.258046 29438 hybrid_clock.cc:648] HybridClock initialized: now 1790330611258019 us; error 44 us; skew 500 ppm
I20260925 10:03:31.259588 29438 webserver.cc:460] Webserver started at http://127.19.46.193:42457/ using document root <none> and password file <none>
I20260925 10:03:31.259866 29438 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:31.259939 29438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:31.260095 29438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:31.261214 29438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "33867a31c0414937bf12971dd9f657dc"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "3836110ff59f395c6baad95be16b4ee2"
server_key_iv: "5b3f118be68c6222847eeffbd362071e"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.261651 29438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "33867a31c0414937bf12971dd9f657dc"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "3836110ff59f395c6baad95be16b4ee2"
server_key_iv: "5b3f118be68c6222847eeffbd362071e"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.263109 29438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.264006 29452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.264282 29438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.264400 29438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "33867a31c0414937bf12971dd9f657dc"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "3836110ff59f395c6baad95be16b4ee2"
server_key_iv: "5b3f118be68c6222847eeffbd362071e"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.264487 29438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:31.269248 29438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:31.269598 29438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:31.269732 29438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:31.269936 29438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:31.270273 29438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:31.270313 29438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.270342 29438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:31.270360 29438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.279153 29438 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:42097
I20260925 10:03:31.279366 29565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:42097 every 8 connection(s)
I20260925 10:03:31.279994 29438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:31.283651 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.280920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:42621 (local address 127.19.46.254:38785)
0925 10:03:31.280981 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:31.280983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:31.281324 (+ 341us) server_negotiation.cc:407] Connection header received
0925 10:03:31.281984 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:31.281985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:31.281992 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:31.282034 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:31.283388 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:31.283391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:31.283391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:31.283396 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:31.283412 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:31.283417 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:31.283470 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:31.283556 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:03:31.283573 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:03:31.283892 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.280904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38785 (local address 127.19.46.193:42621)
0925 10:03:31.281136 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:31.281208 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:31.281857 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:31.282137 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:31.282142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:31.282159 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:31.283216 (+ 1057us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:31.283217 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:31.283249 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:31.283280 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:31.283467 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:31.283472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:31.283475 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:31.283556 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:03:31.283626 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260925 10:03:31.284683 29566 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38785
I20260925 10:03:31.284783 29566 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:31.284955 29566 heartbeater.cc:511] Master 127.19.46.254:38785 requested a full tablet report, sending...
I20260925 10:03:31.286046 29378 ts_manager.cc:194] Registered new tserver with Master: 33867a31c0414937bf12971dd9f657dc (127.19.46.193:42097)
I20260925 10:03:31.288797 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29438
I20260925 10:03:31.289021 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:31.289230 19643 external_mini_cluster.cc:1428] Setting key 121c3b25dfb513764180f371cb4164c8
I20260925 10:03:31.290570 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--builtin_ntp_servers=127.19.46.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:03:31.379240 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:31.379505 29569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:31.379534 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:31.381428 29569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:31.381561 29569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:31.381594 29569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:31.383169 29569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:31.383603 29569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:31.383976 29569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:31.389366 29575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:31.389516 29569 server_base.cc:1034] running on GCE node
W20260925 10:03:31.389366 29574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.389400 29577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:31.389935 29569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:31.390203 29569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:31.391377 29569 hybrid_clock.cc:648] HybridClock initialized: now 1790330611391363 us; error 32 us; skew 500 ppm
I20260925 10:03:31.392962 29569 webserver.cc:460] Webserver started at http://127.19.46.194:40571/ using document root <none> and password file <none>
I20260925 10:03:31.393245 29569 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:31.393294 29569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:31.393465 29569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:31.394587 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "eed8b777fe38443e80346e6f90c7a2a0"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "944bd4fc98a5bc6f4fe4e28217430b85"
server_key_iv: "29e3740d415505961f37e00365565f0a"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.394953 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "eed8b777fe38443e80346e6f90c7a2a0"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "944bd4fc98a5bc6f4fe4e28217430b85"
server_key_iv: "29e3740d415505961f37e00365565f0a"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.396445 29569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:03:31.397347 29583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.397586 29569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.397722 29569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "eed8b777fe38443e80346e6f90c7a2a0"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "944bd4fc98a5bc6f4fe4e28217430b85"
server_key_iv: "29e3740d415505961f37e00365565f0a"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.397807 29569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:31.401232 29569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:31.401574 29569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:31.401727 29569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:31.402263 29569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:31.402853 29569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:31.402894 29569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.402916 29569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:31.402954 29569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.409268 29569 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:38141
I20260925 10:03:31.409359 29697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:38141 every 8 connection(s)
I20260925 10:03:31.409730 29569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:31.413017 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.410560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57639 (local address 127.19.46.254:38785)
0925 10:03:31.410631 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:31.410634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:31.411002 (+ 368us) server_negotiation.cc:407] Connection header received
0925 10:03:31.411492 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:31.411494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:31.411499 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:31.411570 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:31.412757 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:31.412758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:31.412759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:31.412763 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:31.412787 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:31.412791 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:31.412844 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:31.412945 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:03:31.412958 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:03:31.413188 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.410557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38785 (local address 127.19.46.194:57639)
0925 10:03:31.410849 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:31.410887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:31.411393 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:31.411630 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:31.411634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:31.411642 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:31.412599 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:31.412599 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:31.412628 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:31.412658 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:31.412843 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:31.412845 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:31.412848 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:31.412906 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:03:31.412954 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
I20260925 10:03:31.413812 29698 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38785
I20260925 10:03:31.413911 29698 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:31.414067 29698 heartbeater.cc:511] Master 127.19.46.254:38785 requested a full tablet report, sending...
I20260925 10:03:31.414460 29378 ts_manager.cc:194] Registered new tserver with Master: eed8b777fe38443e80346e6f90c7a2a0 (127.19.46.194:38141)
I20260925 10:03:31.416117 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29569
I20260925 10:03:31.416198 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:31.416358 19643 external_mini_cluster.cc:1428] Setting key be61fed6b28f964565cec8a83d6921af
I20260925 10:03:31.417213 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--builtin_ntp_servers=127.19.46.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:03:31.498618 29701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:31.498891 29701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:31.498932 29701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:31.500218 29701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:31.500281 29701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:31.500401 29701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:31.502022 29701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:31.502310 29701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:31.502619 29701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:31.507994 29706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.508025 29709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:31.507999 29707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:31.508304 29701 server_base.cc:1034] running on GCE node
I20260925 10:03:31.508515 29701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:31.508736 29701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:31.509907 29701 hybrid_clock.cc:648] HybridClock initialized: now 1790330611509890 us; error 36 us; skew 500 ppm
I20260925 10:03:31.511278 29701 webserver.cc:460] Webserver started at http://127.19.46.195:33185/ using document root <none> and password file <none>
I20260925 10:03:31.511626 29701 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:31.511716 29701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:31.511895 29701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:31.513075 29701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "0e8ff70416e4473ea5ac5c98c9c90d74"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "5ba536b4763606383ab33c4ef98381c5"
server_key_iv: "1a01a356bd9aac7d5a7029c97a15307c"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.513602 29701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "0e8ff70416e4473ea5ac5c98c9c90d74"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "5ba536b4763606383ab33c4ef98381c5"
server_key_iv: "1a01a356bd9aac7d5a7029c97a15307c"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.515210 29701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.516516 29715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.516765 29701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:31.516885 29701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "0e8ff70416e4473ea5ac5c98c9c90d74"
format_stamp: "Formatted at 2026-09-25 10:03:31 on dist-test-slave-f9rl"
server_key: "5ba536b4763606383ab33c4ef98381c5"
server_key_iv: "1a01a356bd9aac7d5a7029c97a15307c"
server_key_version: "encryptionkey@0"
I20260925 10:03:31.516997 29701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:31.521274 29701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:31.521765 29701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:31.521937 29701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:31.522210 29701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:31.522579 29701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:31.522642 29701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.522688 29701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:31.522719 29701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:31.529225 29701 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:36303
I20260925 10:03:31.529325 29829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:36303 every 8 connection(s)
I20260925 10:03:31.529716 29701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:03:31.532060 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29701
I20260925 10:03:31.532184 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:31.532411 19643 external_mini_cluster.cc:1428] Setting key 718f1c9e5c1c2c1210991664d3a9abef
I20260925 10:03:31.533226 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.530555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55943 (local address 127.19.46.254:38785)
0925 10:03:31.530665 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:31.530667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:31.530880 (+ 213us) server_negotiation.cc:407] Connection header received
0925 10:03:31.531419 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:31.531420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:31.531427 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:31.531478 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:31.532933 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:31.532935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:31.532936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:03:31.532940 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:31.532958 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:03:31.532964 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:31.533022 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:31.533132 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:03:31.533150 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:03:31.533488 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:31.530506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38785 (local address 127.19.46.195:55943)
0925 10:03:31.530759 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:31.530793 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:31.531300 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:31.531546 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:31.531553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:31.531566 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:31.532740 (+ 1174us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:03:31.532740 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:31.532780 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:03:31.532830 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:31.533035 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:31.533041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:31.533044 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:31.533126 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:03:31.533209 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":129,"threads_started":1}
I20260925 10:03:31.533942 29830 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38785
I20260925 10:03:31.534032 29830 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:31.534142 29830 heartbeater.cc:511] Master 127.19.46.254:38785 requested a full tablet report, sending...
I20260925 10:03:31.534523 29378 ts_manager.cc:194] Registered new tserver with Master: 0e8ff70416e4473ea5ac5c98c9c90d74 (127.19.46.195:36303)
I20260925 10:03:31.544768 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:03:31.547106 29417 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:03:31.546497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.19.46.254:38785)
0925 10:03:31.546596 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:31.546599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:31.546671 (+ 72us) server_negotiation.cc:407] Connection header received
0925 10:03:31.546758 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:31.546759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:31.546764 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:31.546844 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:31.547025 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40474: BlockingRecv error: recv got EOF from 127.0.0.1:40474 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:03:31.547225 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29438
I20260925 10:03:31.551991 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29569
I20260925 10:03:31.556466 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29701
I20260925 10:03:31.563275 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29347
2026-09-25T10:03:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (575 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:31 dist-test-slave-f9rl krb5kdc[29840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:31 dist-test-slave-f9rl krb5kdc[29840](info): set up 2 sockets
Sep 25 10:03:31 dist-test-slave-f9rl krb5kdc[29840](info): commencing operation
krb5kdc: starting...
W20260925 10:03:32.639999 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:32 dist-test-slave-f9rl krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:03:32.768982 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40955
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:40955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:32.856184 29853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:32.856451 29853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:32.856503 29853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:32.857774 29853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:03:32.857851 29853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:32.857877 29853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:32.857898 29853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:03:32.857926 29853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:03:32.859526 29853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:40955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:32.859840 29853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:32.860093 29853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:32.865427 29858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:32.865427 29859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:32.865453 29853 server_base.cc:1034] running on GCE node
W20260925 10:03:32.865496 29861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:32.866398 29853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:32.866739 29853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:32.867937 29853 hybrid_clock.cc:648] HybridClock initialized: now 1790330612867920 us; error 34 us; skew 500 ppm
Sep 25 10:03:32 dist-test-slave-f9rl krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330612, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:32.870986 29853 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:03:32.871513 29853 webserver.cc:460] Webserver started at http://127.19.46.254:42917/ using document root <none> and password file <none>
I20260925 10:03:32.871757 29853 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:32.871829 29853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:32.871987 29853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:32.873107 29853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "92fdd1ec0f644bc5a911f966245b655a"
format_stamp: "Formatted at 2026-09-25 10:03:32 on dist-test-slave-f9rl"
server_key: "3852dc6dcd615ce4be42ccb7d7599757"
server_key_iv: "2e883c50e77cf6798caeed0461ceb1e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:32.873544 29853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "92fdd1ec0f644bc5a911f966245b655a"
format_stamp: "Formatted at 2026-09-25 10:03:32 on dist-test-slave-f9rl"
server_key: "3852dc6dcd615ce4be42ccb7d7599757"
server_key_iv: "2e883c50e77cf6798caeed0461ceb1e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:32.874980 29853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:32.875834 29869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:32.876034 29853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:32.876147 29853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "92fdd1ec0f644bc5a911f966245b655a"
format_stamp: "Formatted at 2026-09-25 10:03:32 on dist-test-slave-f9rl"
server_key: "3852dc6dcd615ce4be42ccb7d7599757"
server_key_iv: "2e883c50e77cf6798caeed0461ceb1e2"
server_key_version: "encryptionkey@0"
I20260925 10:03:32.876246 29853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:32.888106 29853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:32.890627 29853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:32.890815 29853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:32.894939 29853 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:40955
I20260925 10:03:32.894994 29921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:40955 every 8 connection(s)
I20260925 10:03:32.895390 29853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:03:32.896598 29922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:32.899180 29922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: Bootstrap starting.
I20260925 10:03:32.899821 29922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:32.900138 29922 log.cc:824] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:32.900959 29922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: No bootstrap required, opened a new log
I20260925 10:03:32.902716 29922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } }
I20260925 10:03:32.902868 29922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:32.902892 29922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fdd1ec0f644bc5a911f966245b655a, State: Initialized, Role: FOLLOWER
I20260925 10:03:32.903059 29922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } }
I20260925 10:03:32.903141 29922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:03:32.903163 29922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:03:32.903234 29922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:32.904188 29922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } }
I20260925 10:03:32.904292 29922 leader_election.cc:302] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92fdd1ec0f644bc5a911f966245b655a; no voters:
I20260925 10:03:32.904491 29922 leader_election.cc:288] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:03:32.904675 29925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:32.904901 29925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [term 1 LEADER]: Becoming Leader. State: Replica: 92fdd1ec0f644bc5a911f966245b655a, State: Running, Role: LEADER
I20260925 10:03:32.904923 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29853
I20260925 10:03:32.904978 29922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:03:32.905042 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:03:32.905079 29925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } }
I20260925 10:03:32.905251 19643 external_mini_cluster.cc:1428] Setting key 1278f647e74b76ce9468e69dfd73bd7d
I20260925 10:03:32.905493 29927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92fdd1ec0f644bc5a911f966245b655a. Latest consensus state: current_term: 1 leader_uuid: "92fdd1ec0f644bc5a911f966245b655a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } } }
I20260925 10:03:32.905577 29927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:32.905854 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92fdd1ec0f644bc5a911f966245b655a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fdd1ec0f644bc5a911f966245b655a" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40955 } } }
I20260925 10:03:32.905931 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a [sys.catalog]: This master's current role is: LEADER
I20260925 10:03:32.905966 29934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:03:32.906760 29934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:03:32.908471 29934 catalog_manager.cc:1347] Generated new cluster ID: af40c674af6f434f9acb2156a2989c8b
I20260925 10:03:32.908525 29934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:03:32 dist-test-slave-f9rl krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:32.917851 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:32.905930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.19.46.254:40955)
0925 10:03:32.906257 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:32.906328 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:32.906373 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:03:32.906514 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:32.906522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:32.906599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:32.906766 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:32.907571 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:32.909087 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:32.910178 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:32.910848 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:32.910968 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:32.913895 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:32.913900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:32.913900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:32.913915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:32.916027 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:32.916476 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:32.916479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:32.916479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:32.916605 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:32.916824 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:32.916825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:32.916825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:32.917153 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:32.917320 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:32.917453 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:03:32.917601 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":80,"threads_started":1}
I20260925 10:03:32.932447 29934 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:03:32.933063 29934 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:03:32.938287 29934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: Generated new TSK 0
I20260925 10:03:32.938489 29934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:03:32.977380 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:33.067206 29948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:33.067484 29948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:33.067510 29948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:33.068737 29948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:33.068783 29948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:33.068810 29948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:03:33.070416 29948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:33.070683 29948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:33.070937 29948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:33.076454 29953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.076465 29956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.076465 29954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:33.077042 29948 server_base.cc:1034] running on GCE node
I20260925 10:03:33.077312 29948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:33.077642 29948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:33.078796 29948 hybrid_clock.cc:648] HybridClock initialized: now 1790330613078769 us; error 49 us; skew 500 ppm
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:33.081799 29948 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:03:33.082304 29948 webserver.cc:460] Webserver started at http://127.19.46.193:43717/ using document root <none> and password file <none>
I20260925 10:03:33.082576 29948 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:33.082647 29948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:33.082798 29948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:33.083905 29948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "40425331504d44ec954918965f548e21"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "c65795eb7597753381d864e009490131"
server_key_iv: "1cacef4af36f74618a79fafe2e20d3e0"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.084308 29948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "40425331504d44ec954918965f548e21"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "c65795eb7597753381d864e009490131"
server_key_iv: "1cacef4af36f74618a79fafe2e20d3e0"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.085965 29948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:03:33.086982 29963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.087579 29948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:33.087781 29948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "40425331504d44ec954918965f548e21"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "c65795eb7597753381d864e009490131"
server_key_iv: "1cacef4af36f74618a79fafe2e20d3e0"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.087872 29948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:33.099152 29948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:33.101706 29948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:33.101883 29948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:33.102139 29948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:33.102464 29948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:33.102517 29948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.102558 29948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:33.102588 29948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.110461 29948 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:33605
I20260925 10:03:33.110566 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:33605 every 8 connection(s)
I20260925 10:03:33.110915 29948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:03:33.113124 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 29948
I20260925 10:03:33.113255 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:03:33.113502 19643 external_mini_cluster.cc:1428] Setting key ec7dbfc15fbd5f19abf24eca23632b1b
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:03:33.121279 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.111735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:44283 (local address 127.19.46.254:40955)
0925 10:03:33.111807 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.111809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.112024 (+ 215us) server_negotiation.cc:407] Connection header received
0925 10:03:33.112520 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.112521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.112534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.112588 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:33.113991 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.114643 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.115664 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.115899 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.115989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.118306 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:33.118307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:33.118308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:33.118316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:33.119791 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.120234 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.120235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.120236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.120296 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.120561 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.120562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.120562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.120691 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:33.120763 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.121056 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.121181 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:03:33.121480 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.111723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40955 (local address 127.19.46.193:44283)
0925 10:03:33.111901 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.111939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.112432 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.112643 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.112646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.113520 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:33.113890 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.113897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.114745 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.114746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.115564 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.115566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.116000 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.116001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.116148 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.116766 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:33.116775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:33.118201 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:33.119901 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.119910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.119912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.120143 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.120370 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.120371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.120371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.120486 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.120766 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:33.120774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:33.120920 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.121033 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.121200 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260925 10:03:33.122112 30078 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40955
I20260925 10:03:33.122475 30078 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:33.122725 30078 heartbeater.cc:511] Master 127.19.46.254:40955 requested a full tablet report, sending...
I20260925 10:03:33.123517 29886 ts_manager.cc:194] Registered new tserver with Master: 40425331504d44ec954918965f548e21 (127.19.46.193:33605)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
I20260925 10:03:33.124352 29886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:44283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:03:33.141812 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:33.225143 30083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:33.225366 30083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:33.225391 30083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:33.226584 30083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:33.226634 30083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:33.226663 30083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:03:33.228092 30083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:33.228296 30083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:33.228477 30083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:33.233697 30088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.233690 30089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.233691 30091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:33.233708 30083 server_base.cc:1034] running on GCE node
I20260925 10:03:33.234161 30083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:33.234421 30083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:33.235627 30083 hybrid_clock.cc:648] HybridClock initialized: now 1790330613235581 us; error 30 us; skew 500 ppm
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:33.238243 30083 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:03:33.238812 30083 webserver.cc:460] Webserver started at http://127.19.46.194:45025/ using document root <none> and password file <none>
I20260925 10:03:33.239048 30083 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:33.239122 30083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:33.239279 30083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:33.240388 30083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "1bde6b0ebe8346e99d4cd1c863c12937"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "a2d855a02d8777591d4389d5eb62b2bc"
server_key_iv: "02231b887de3e5223e0298623d5c562b"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.240770 30083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "1bde6b0ebe8346e99d4cd1c863c12937"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "a2d855a02d8777591d4389d5eb62b2bc"
server_key_iv: "02231b887de3e5223e0298623d5c562b"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.242249 30083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:03:33.243173 30098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.243419 30083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:03:33.243547 30083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "1bde6b0ebe8346e99d4cd1c863c12937"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "a2d855a02d8777591d4389d5eb62b2bc"
server_key_iv: "02231b887de3e5223e0298623d5c562b"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.243628 30083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:33.257150 30083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:33.259322 30083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:33.259516 30083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:33.259766 30083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:33.260090 30083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:33.260144 30083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.260180 30083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:33.260210 30083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.266470 30083 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:42319
I20260925 10:03:33.266515 30211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:42319 every 8 connection(s)
I20260925 10:03:33.266852 30083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:03:33.267058 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 30083
I20260925 10:03:33.267174 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:03:33.267457 19643 external_mini_cluster.cc:1428] Setting key 88f27f8a07ad5d733769a3ffc1489896
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:03:33.277964 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.267770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:40037 (local address 127.19.46.254:40955)
0925 10:03:33.267847 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.267850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.268040 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:03:33.268641 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.268642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.268654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.268700 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:33.269795 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.270321 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.271895 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.272166 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.272245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.274780 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:33.274781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:33.274782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:33.274790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:33.276480 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.276933 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.276935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.276935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.276999 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.277264 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.277265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.277266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.277396 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:33.277492 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.277772 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.277866 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:03:33.278172 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.267671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40955 (local address 127.19.46.194:40037)
0925 10:03:33.267897 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.267947 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.268534 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.268775 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.268779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.269375 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:33.269678 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.269683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.270431 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.270432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.271770 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.271774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.272273 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.272274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.272434 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.272984 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:33.272995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:33.274669 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:33.276584 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.276593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.276596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.276841 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.277069 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.277070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.277070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.277192 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.277480 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:33.277486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:33.277639 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.277752 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.277919 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260925 10:03:33.278712 30212 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40955
I20260925 10:03:33.278961 30212 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
I20260925 10:03:33.279191 30212 heartbeater.cc:511] Master 127.19.46.254:40955 requested a full tablet report, sending...
I20260925 10:03:33.279637 29886 ts_manager.cc:194] Registered new tserver with Master: 1bde6b0ebe8346e99d4cd1c863c12937 (127.19.46.194:42319)
I20260925 10:03:33.280040 29886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:40037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:03:33.296703 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:03:33.378007 30217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:03:33.378240 30217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:03:33.378268 30217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:03:33.379513 30217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:03:33.379560 30217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:03:33.379590 30217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:03:33.381294 30217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:03:33.381580 30217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:03:33.381783 30217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:03:33.387087 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.387071 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:03:33.387071 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:03:33.387210 30217 server_base.cc:1034] running on GCE node
I20260925 10:03:33.387614 30217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:03:33.387866 30217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:03:33.389020 30217 hybrid_clock.cc:648] HybridClock initialized: now 1790330613388988 us; error 53 us; skew 500 ppm
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:03:33.391979 30217 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:03:33.392417 30217 webserver.cc:460] Webserver started at http://127.19.46.195:44253/ using document root <none> and password file <none>
I20260925 10:03:33.392633 30217 fs_manager.cc:359] Metadata directory not provided
I20260925 10:03:33.392676 30217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:03:33.392769 30217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:03:33.393919 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "24149d42762f4928a0f3043330d6cda9"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "7a87c3bfff647f4bc1a352093978e306"
server_key_iv: "67be3c608eee2410eeb0b89914c3c440"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.394230 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "24149d42762f4928a0f3043330d6cda9"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "7a87c3bfff647f4bc1a352093978e306"
server_key_iv: "67be3c608eee2410eeb0b89914c3c440"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.395547 30217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:33.396306 30232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.396521 30217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:03:33.396651 30217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "24149d42762f4928a0f3043330d6cda9"
format_stamp: "Formatted at 2026-09-25 10:03:33 on dist-test-slave-f9rl"
server_key: "7a87c3bfff647f4bc1a352093978e306"
server_key_iv: "67be3c608eee2410eeb0b89914c3c440"
server_key_version: "encryptionkey@0"
I20260925 10:03:33.396739 30217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:03:33.433210 30217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:03:33.435817 30217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:03:33.436026 30217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:03:33.436319 30217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:03:33.436683 30217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:03:33.436746 30217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.436796 30217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:03:33.436846 30217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:03:33.443943 30217 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:45461
I20260925 10:03:33.443991 30345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:45461 every 8 connection(s)
I20260925 10:03:33.444371 30217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330613, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:03:33.452521 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 30217
I20260925 10:03:33.452636 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:03:33.452826 19643 external_mini_cluster.cc:1428] Setting key 50ade995d54e5561eb8978231352c92c
I20260925 10:03:33.455940 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.445193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:53061 (local address 127.19.46.254:40955)
0925 10:03:33.445265 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.445267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.445522 (+ 255us) server_negotiation.cc:407] Connection header received
0925 10:03:33.446645 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.446646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.446661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.446726 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:33.447902 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.448521 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.449663 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.449874 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.449944 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.452352 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:33.452354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:33.452354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:33.452363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:33.454218 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.454681 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.454682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.454683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.454748 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.454988 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.454989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.454989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.455136 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:33.455207 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.455764 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.455845 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:03:33.456450 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.445171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40955 (local address 127.19.46.195:53061)
0925 10:03:33.445339 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.445387 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.446332 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.446814 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.446818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.447445 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:03:33.447656 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.447662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.448646 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.448650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.449571 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.449574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.449986 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.449988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.450115 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.450669 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:03:33.450679 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:03:33.452245 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:03:33.454314 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.454322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.454324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.454583 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.454822 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:03:33.454823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:03:33.454823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:03:33.454926 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:03:33.455299 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:03:33.455313 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:03:33.455553 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.455720 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.455942 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260925 10:03:33.457542 30346 heartbeater.cc:347] Connected to a master server at 127.19.46.254:40955
I20260925 10:03:33.457676 30346 heartbeater.cc:464] Registering TS with master...
I20260925 10:03:33.457952 30346 heartbeater.cc:511] Master 127.19.46.254:40955 requested a full tablet report, sending...
I20260925 10:03:33.458432 29886 ts_manager.cc:194] Registered new tserver with Master: 24149d42762f4928a0f3043330d6cda9 (127.19.46.195:45461)
I20260925 10:03:33.458937 29886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:53061
I20260925 10:03:33.465205 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:03:33.474154 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.466698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.19.46.254:40955)
0925 10:03:33.466777 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.466778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.466822 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:03:33.466893 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.466893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.466902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.466943 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:03:33.467563 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.468245 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.469123 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.469413 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.469629 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.470964 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:03:33.470965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:03:33.470966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:03:33.470974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:03:33.472780 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.473275 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.473276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.473277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.473336 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:03:33.473573 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:03:33.473574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:03:33.473574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:03:33.473762 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:03:33.473858 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.473965 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.474036 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:03:33.476100 29886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:03:33.477888 29886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:03:33.485579 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.19.46.193:33605)
0925 10:03:33.481468 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.481474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.481694 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:03:33.481909 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.481911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.481927 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.481996 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:33.482202 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.483320 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.484563 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.485060 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.485123 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.485169 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.485408 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.485484 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 10:03:33.485569 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33605 (local address 127.0.0.1:36266)
0925 10:03:33.481275 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.481626 (+ 351us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.481724 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.482066 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.482067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.482068 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:33.482103 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.482105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.483427 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.483428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.484433 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.484436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.485169 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.485170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.485270 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.485285 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.485369 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.485432 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260925 10:03:33.485730 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:42319 (local address 127.0.0.1:53702)
0925 10:03:33.481474 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.481513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.481585 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.482003 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.482004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.482008 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:33.482067 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.482068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.483056 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.483057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.484394 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.484397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.485489 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.485489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.485559 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.485562 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.485614 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.485658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":25,"thread_start_us":87,"threads_started":1}
I20260925 10:03:33.485958 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.19.46.194:42319)
0925 10:03:33.481743 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.481746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.481778 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:03:33.481819 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.481821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.481837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.481915 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:33.482147 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.482952 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.484503 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.485060 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.485177 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.485241 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.485837 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.485893 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":89,"threads_started":1}
I20260925 10:03:33.486410 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:45461 (local address 127.0.0.1:43570)
0925 10:03:33.481507 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.481517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.481585 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.481913 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.481917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.481919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:33.482004 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.482006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.483394 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.483395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.484507 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.484508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.486147 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.486148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.486220 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.486225 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.486292 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.486337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":58,"threads_started":1}
I20260925 10:03:33.486733 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.481439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.19.46.195:45461)
0925 10:03:33.481644 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.481649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.481659 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:03:33.481704 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.481706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.481724 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.481803 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:33.482087 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.483279 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.485203 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.485989 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.486059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.486110 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.486590 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.486642 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20260925 10:03:33.486754 30146 tablet_service.cc:1467] Processing CreateTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d (DEFAULT_TABLE table=test-table [id=597a4e31b0fe444b84b8811747d660ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:33.486788 30012 tablet_service.cc:1467] Processing CreateTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d (DEFAULT_TABLE table=test-table [id=597a4e31b0fe444b84b8811747d660ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:33.487178 30146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189cfaf6ef7049f4965e0759faf5b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:33.487177 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189cfaf6ef7049f4965e0759faf5b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:33.487356 30280 tablet_service.cc:1467] Processing CreateTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d (DEFAULT_TABLE table=test-table [id=597a4e31b0fe444b84b8811747d660ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:03:33.487668 30280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189cfaf6ef7049f4965e0759faf5b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:03:33.490274 30363 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Bootstrap starting.
I20260925 10:03:33.490770 30364 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Bootstrap starting.
I20260925 10:03:33.491255 30363 tablet_bootstrap.cc:654] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:33.491238 30365 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Bootstrap starting.
I20260925 10:03:33.491569 30364 tablet_bootstrap.cc:654] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:33.491926 30363 log.cc:824] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:33.491964 30365 tablet_bootstrap.cc:654] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Neither blocks nor log segments found. Creating new log.
I20260925 10:03:33.492161 30364 log.cc:824] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:33.492605 30365 log.cc:824] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:03:33.492834 30363 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: No bootstrap required, opened a new log
I20260925 10:03:33.492911 30363 ts_tablet_manager.cc:1397] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:03:33.493351 30365 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: No bootstrap required, opened a new log
I20260925 10:03:33.493407 30365 ts_tablet_manager.cc:1397] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:03:33.493978 30364 tablet_bootstrap.cc:492] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: No bootstrap required, opened a new log
I20260925 10:03:33.494096 30364 ts_tablet_manager.cc:1397] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:03:33.494872 30365 raft_consensus.cc:348] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.495046 30365 raft_consensus.cc:374] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:33.495111 30365 raft_consensus.cc:729] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24149d42762f4928a0f3043330d6cda9, State: Initialized, Role: FOLLOWER
I20260925 10:03:33.495046 30363 raft_consensus.cc:348] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.495222 30363 raft_consensus.cc:374] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:33.495267 30363 raft_consensus.cc:729] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bde6b0ebe8346e99d4cd1c863c12937, State: Initialized, Role: FOLLOWER
I20260925 10:03:33.495275 30365 consensus_queue.cc:260] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.495375 30363 consensus_queue.cc:260] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.495576 30365 ts_tablet_manager.cc:1428] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:03:33.495584 30346 heartbeater.cc:503] Master 127.19.46.254:40955 was elected leader, sending a full tablet report...
I20260925 10:03:33.495662 30363 ts_tablet_manager.cc:1428] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:33.495826 30212 heartbeater.cc:503] Master 127.19.46.254:40955 was elected leader, sending a full tablet report...
I20260925 10:03:33.496294 30364 raft_consensus.cc:348] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.496503 30364 raft_consensus.cc:374] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:03:33.496578 30364 raft_consensus.cc:729] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40425331504d44ec954918965f548e21, State: Initialized, Role: FOLLOWER
I20260925 10:03:33.496726 30364 consensus_queue.cc:260] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.497049 30364 ts_tablet_manager.cc:1428] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:03:33.497108 30078 heartbeater.cc:503] Master 127.19.46.254:40955 was elected leader, sending a full tablet report...
W20260925 10:03:33.517613 30213 tablet.cc:2369] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:03:33.611788 30079 tablet.cc:2369] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:03:33.694998 30347 tablet.cc:2369] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:03:33.699908 30369 raft_consensus.cc:482] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:03:33.700099 30369 raft_consensus.cc:504] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.700546 30369 leader_election.cc:288] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bde6b0ebe8346e99d4cd1c863c12937 (127.19.46.194:42319), 40425331504d44ec954918965f548e21 (127.19.46.193:33605)
I20260925 10:03:33.704669 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.700864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:57821 (local address 127.19.46.193:33605)
0925 10:03:33.700928 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.700930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.700970 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:03:33.701093 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.701094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.701101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.701170 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:33.701335 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.702140 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.703745 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.704297 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.704378 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.704387 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.704554 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.704592 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:03:33.704669 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.700850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:44067 (local address 127.19.46.194:42319)
0925 10:03:33.700916 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.700919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.701113 (+ 194us) server_negotiation.cc:407] Connection header received
0925 10:03:33.701200 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.701201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.701212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.701282 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:03:33.701468 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.702155 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.703739 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.704306 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.704379 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.704387 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.704554 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.704593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:03:33.704718 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.700816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:42319 (local address 127.19.46.195:44067)
0925 10:03:33.701055 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.701064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.701118 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.701294 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.701295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.701297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:33.701368 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.701369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.702237 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.702238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.703638 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.703641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.704390 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.704391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.704470 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.704475 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.704535 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.704590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":158,"thread_start_us":122,"threads_started":1}
I20260925 10:03:33.705052 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.700814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33605 (local address 127.19.46.195:57821)
0925 10:03:33.700900 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.700915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:03:33.701013 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:03:33.701158 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:03:33.701159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:03:33.701161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:03:33.701242 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.701244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.702237 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.702238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.703638 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:03:33.703641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.704389 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.704391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:03:33.704474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.704475 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:03:33.704535 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:03:33.704590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":69}
I20260925 10:03:33.705127 30166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189cfaf6ef7049f4965e0759faf5b47d" candidate_uuid: "24149d42762f4928a0f3043330d6cda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" is_pre_election: true
I20260925 10:03:33.705368 30166 raft_consensus.cc:2454] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24149d42762f4928a0f3043330d6cda9 in term 0.
I20260925 10:03:33.705480 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189cfaf6ef7049f4965e0759faf5b47d" candidate_uuid: "24149d42762f4928a0f3043330d6cda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40425331504d44ec954918965f548e21" is_pre_election: true
I20260925 10:03:33.705631 30032 raft_consensus.cc:2454] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24149d42762f4928a0f3043330d6cda9 in term 0.
I20260925 10:03:33.705680 30233 leader_election.cc:302] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bde6b0ebe8346e99d4cd1c863c12937, 24149d42762f4928a0f3043330d6cda9; no voters:
I20260925 10:03:33.705818 30369 raft_consensus.cc:2781] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:03:33.705889 30369 raft_consensus.cc:482] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:03:33.705912 30369 raft_consensus.cc:3037] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:33.706761 30369 raft_consensus.cc:504] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.706913 30369 leader_election.cc:288] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [CANDIDATE]: Term 1 election: Requested vote from peers 1bde6b0ebe8346e99d4cd1c863c12937 (127.19.46.194:42319), 40425331504d44ec954918965f548e21 (127.19.46.193:33605)
I20260925 10:03:33.707075 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189cfaf6ef7049f4965e0759faf5b47d" candidate_uuid: "24149d42762f4928a0f3043330d6cda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40425331504d44ec954918965f548e21"
I20260925 10:03:33.707069 30166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189cfaf6ef7049f4965e0759faf5b47d" candidate_uuid: "24149d42762f4928a0f3043330d6cda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bde6b0ebe8346e99d4cd1c863c12937"
I20260925 10:03:33.707154 30032 raft_consensus.cc:3037] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:33.707151 30166 raft_consensus.cc:3037] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:03:33.707876 30032 raft_consensus.cc:2454] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24149d42762f4928a0f3043330d6cda9 in term 1.
I20260925 10:03:33.707921 30166 raft_consensus.cc:2454] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24149d42762f4928a0f3043330d6cda9 in term 1.
I20260925 10:03:33.708091 30235 leader_election.cc:302] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24149d42762f4928a0f3043330d6cda9, 40425331504d44ec954918965f548e21; no voters:
I20260925 10:03:33.708216 30369 raft_consensus.cc:2781] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:03:33.708439 30369 raft_consensus.cc:686] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 1 LEADER]: Becoming Leader. State: Replica: 24149d42762f4928a0f3043330d6cda9, State: Running, Role: LEADER
I20260925 10:03:33.708541 30369 consensus_queue.cc:237] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } }
I20260925 10:03:33.709347 29885 catalog_manager.cc:5495] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24149d42762f4928a0f3043330d6cda9 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "24149d42762f4928a0f3043330d6cda9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24149d42762f4928a0f3043330d6cda9" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20260925 10:03:33.770064 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.767508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.19.46.195:45461)
0925 10:03:33.767634 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 10:03:33.767636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:03:33.767645 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:03:33.767682 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:03:33.767683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:03:33.767701 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:03:33.767769 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:03:33.767932 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.768484 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.769300 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:03:33.769506 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:03:33.769564 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:03:33.769764 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:03:33.769866 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:03:33.769929 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:03:33.769968 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 10:03:33.769985 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 10:03:33.771860 30032 raft_consensus.cc:1261] T 189cfaf6ef7049f4965e0759faf5b47d P 40425331504d44ec954918965f548e21 [term 1 FOLLOWER]: Refusing update from remote peer 24149d42762f4928a0f3043330d6cda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:33.771869 30166 raft_consensus.cc:1261] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 1 FOLLOWER]: Refusing update from remote peer 24149d42762f4928a0f3043330d6cda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:03:33.772269 30373 consensus_queue.cc:1035] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40425331504d44ec954918965f548e21" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:33.772408 30369 consensus_queue.cc:1035] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bde6b0ebe8346e99d4cd1c863c12937" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:03:33.774793 30376 mvcc.cc:205] Tried to move back new op lower bound from 7333194194006663168 to 7333194193750728704. Current Snapshot: MvccSnapshot[applied={T|T < 7333194194006663168}]
I20260925 10:03:33.775184 30378 mvcc.cc:205] Tried to move back new op lower bound from 7333194194006663168 to 7333194193750728704. Current Snapshot: MvccSnapshot[applied={T|T < 7333194194006663168}]
I20260925 10:03:33.775403 30377 mvcc.cc:205] Tried to move back new op lower bound from 7333194194006663168 to 7333194193750728704. Current Snapshot: MvccSnapshot[applied={T|T < 7333194194006663168}]
I20260925 10:03:33.776798 29884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:33.776902 29884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:03:33.777717 29884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92fdd1ec0f644bc5a911f966245b655a: Sending DeleteTablet for 3 replicas of tablet 189cfaf6ef7049f4965e0759faf5b47d
I20260925 10:03:33.778157 30012 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:33 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:36266
I20260925 10:03:33.778205 30280 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:33 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:43570
I20260925 10:03:33.778183 30146 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cfaf6ef7049f4965e0759faf5b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:03:33 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:53702
I20260925 10:03:33.778393 30387 tablet_replica.cc:330] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: stopping tablet replica
I20260925 10:03:33.778527 30387 raft_consensus.cc:2229] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:03:33.778586 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29948
I20260925 10:03:33.778697 30387 raft_consensus.cc:2258] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:33.779297 30387 ts_tablet_manager.cc:1905] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:33.781136 30387 ts_tablet_manager.cc:1918] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:33.781278 30387 log.cc:1196] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/189cfaf6ef7049f4965e0759faf5b47d
I20260925 10:03:33.781646 30387 ts_tablet_manager.cc:1939] T 189cfaf6ef7049f4965e0759faf5b47d P 1bde6b0ebe8346e99d4cd1c863c12937: Deleting consensus metadata
I20260925 10:03:33.782197 29870 catalog_manager.cc:4841] TS 1bde6b0ebe8346e99d4cd1c863c12937 (127.19.46.194:42319): tablet 189cfaf6ef7049f4965e0759faf5b47d (table test-table [id=597a4e31b0fe444b84b8811747d660ad]) successfully deleted
I20260925 10:03:33.785571 30388 tablet_replica.cc:330] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: stopping tablet replica
I20260925 10:03:33.785698 30388 raft_consensus.cc:2229] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:03:33.785801 30388 raft_consensus.cc:2258] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:03:33.786231 30388 ts_tablet_manager.cc:1905] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:03:33.787946 30388 ts_tablet_manager.cc:1918] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:03:33.788023 30388 log.cc:1196] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/189cfaf6ef7049f4965e0759faf5b47d
I20260925 10:03:33.788291 30388 ts_tablet_manager.cc:1939] T 189cfaf6ef7049f4965e0759faf5b47d P 24149d42762f4928a0f3043330d6cda9: Deleting consensus metadata
I20260925 10:03:33.788753 29870 catalog_manager.cc:4841] TS 24149d42762f4928a0f3043330d6cda9 (127.19.46.195:45461): tablet 189cfaf6ef7049f4965e0759faf5b47d (table test-table [id=597a4e31b0fe444b84b8811747d660ad]) successfully deleted
W20260925 10:03:33.789371 29872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:33605 (error 108)
I20260925 10:03:33.789552 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 30083
I20260925 10:03:33.790441 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:03:33.790282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33605 (local address 127.0.0.1:36278)
0925 10:03:33.790352 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 10:03:33.790399 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:03:33.790578 29872 catalog_manager.cc:4584] TS 40425331504d44ec954918965f548e21 (127.19.46.193:33605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 189cfaf6ef7049f4965e0759faf5b47d: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33605: connect: Connection refused (error 111)
I20260925 10:03:33.795379 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 30217
I20260925 10:03:33.800122 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 29853
2026-09-25T10:03:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2235 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[30392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[30392](info): set up 2 sockets
Sep 25 10:03:33 dist-test-slave-f9rl krb5kdc[30392](info): commencing operation
krb5kdc: starting...
W20260925 10:03:34.876693 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:03:34 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:03:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangeradmin_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangeradmin_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangerlookup_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangerlookup_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
I20260925 10:03:35.061890 19643 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task46RfQ8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/postgres -l logfile start
2026-09-25 10:03:37.987 UTC [30422] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:03:37.987 UTC [30422] LOG: listening on IPv4 address "127.19.46.212", port 39419
2026-09-25 10:03:37.992 UTC [30422] LOG: listening on Unix socket "/tmp/.s.PGSQL.39419"
2026-09-25 10:03:38.014 UTC [30425] LOG: database system was shut down at 2026-09-25 10:03:36 UTC
2026-09-25 10:03:38.018 UTC [30422] LOG: database system is ready to accept connections
I20260925 10:03:39.016309 19643 mini_postgres.cc:96] Postgres bound to 39419
2026-09-25 10:03:39.023 UTC [30433] FATAL: database "slave" does not exist
127.19.46.212:39419 - accepting connections
I20260925 10:03:39.023699 19643 mini_ranger.cc:161] Starting Ranger...
I20260925 10:03:39.039141 19643 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:03:39.629547 19643 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:03:39.629650 19643 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin
2026-09-25 10:03:39,921 [I] DB FLAVOR :POSTGRES
2026-09-25 10:03:39,921 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:03:39,921 [I] Checking connection..
2026-09-25 10:03:39,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:03:40,228 [I] Checking connection passed.
2026-09-25 10:03:40,228 [I] --------- Verifying version history table ---------
2026-09-25 10:03:40,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:03:40,521 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:03:40,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:03:40,786 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:03:40,787 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:03:40,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:03:41.062 UTC [30498] WARNING: there is no transaction in progress
2026-09-25 10:03:41,070 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:03:41,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:03:41,354 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:03:41,355 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:03:41,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:03:41,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:03:41,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rl.c.gcp-upstream.internal','N') ;"
2026-09-25 10:03:42,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:03:42,468 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:03:42,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:03:42,747 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:03:42,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:03:43,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:03:43,333 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:03:43,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:03:44.450 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.469 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.487 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.505 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.524 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.546 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.621 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.630 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.648 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.662 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.674 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.691 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.702 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.712 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.722 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:44.731 UTC [30607] WARNING: there is no transaction in progress
2026-09-25 10:03:45,588 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:03:45,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:03:45,857 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:03:45,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:03:46,148 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:03:46,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:03:46,427 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:03:46,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:03:46,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:03:46,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rl.c.gcp-upstream.internal';"
2026-09-25 10:03:47,255 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:03:47,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:03:47,539 [I] DB_PATCHES have already been applied
I20260925 10:03:47.544807 19643 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task46RfQ8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:03:47.544911 19643 mini_ranger.cc:193] Using host: 127.19.46.212
I20260925 10:03:47.548242 19643 mini_ranger.cc:227] Ranger admin URL: http://127.19.46.212:41403
Sep 25, 2026 10:03:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:03:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:03:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:03:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:03:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:03:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:03:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:03:48.715042 19643 mini_ranger.cc:160] Time spent starting Ranger: real 9.691s user 0.000s sys 0.009s
Sep 25, 2026 10:04:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:04:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:04:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:04:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:04:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:04:14.080201 19643 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangerkms_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/rangerkms_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:04:14 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330654, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:04:14.188355 19643 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:04:14.201514 19643 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:04:15.369524 19643 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:04:15.369609 19643 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms
I20260925 10:04:15.369632 19643 mini_ranger_kms.cc:227] Using postgres at 127.19.46.212:39419
2026-09-25 10:04:15,876 [I] DB FLAVOR :POSTGRES
2026-09-25 10:04:15,876 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:04:15,877 [I] Checking connection
2026-09-25 10:04:15,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:04:16,155 [I] connection success
2026-09-25 10:04:16,155 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:04:16,155 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:04:16,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:04:16,425 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:04:16,426 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:04:16,426 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:04:16,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:39419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:04:16,761 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:04:17.323993 19643 mini_ranger_kms.cc:310] Created kms service
I20260925 10:04:17.577399 19643 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:04:17.631597 19643 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:04:17 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330654, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:04:18.099182 19643 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:04:18.099309 19643 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms:/tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:04:18.099332 19643 mini_ranger_kms.cc:240] Using host: 127.19.46.212
I20260925 10:04:18.103093 19643 mini_ranger_kms.cc:276] Ranger KMS PID: 30789
I20260925 10:04:18.103250 19643 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.46.212:52789
Sep 25, 2026 10:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:04:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:04:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:04:20.619796 19643 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.431s user 0.003s sys 0.212s
I20260925 10:04:20.619941 19643 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:04:20.620051 19643 mini_ranger_kms.cc:401] 127.19.46.212:52789/kms/v1/keys
Sep 25 10:04:20 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:04:21 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
Sep 25 10:04:22 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
Sep 25 10:04:24 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:04:24.995541 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45023
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:45023
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:52789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:04:25.107863 30840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:04:25.108143 30840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:04:25.108173 30840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:04:25.110186 30840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:04:25.110265 30840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:04:25.110283 30840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:04:25.110298 30840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:04:25.110312 30840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:04:25.113291 30840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.46.212:52789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:45023
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:04:25.113672 30840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:04:25.114014 30840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:04:25.121615 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:25.121834 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:25.121618 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:04:25.122016 30840 server_base.cc:1034] running on GCE node
I20260925 10:04:25.122772 30840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:04:25.123360 30840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:04:25.124591 30840 hybrid_clock.cc:648] HybridClock initialized: now 1790330665124485 us; error 125 us; skew 500 ppm
Sep 25 10:04:25 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330665, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:04:25.144069 30840 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:04:25.144835 30840 webserver.cc:460] Webserver started at http://127.19.46.254:37119/ using document root <none> and password file <none>
I20260925 10:04:25.145328 30840 fs_manager.cc:359] Metadata directory not provided
I20260925 10:04:25.145411 30840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:04:25.145637 30840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:04:25 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330665, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:04:25.252692 30840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "c43afefb05b84a6fa60decfdd03652be"
format_stamp: "Formatted at 2026-09-25 10:04:25 on dist-test-slave-f9rl"
server_key: "726c31d58594dfb7565c037c203146e4"
server_key_iv: "aee58e5f441e57c52eb643972a92eff0"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:25.253340 30840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "c43afefb05b84a6fa60decfdd03652be"
format_stamp: "Formatted at 2026-09-25 10:04:25 on dist-test-slave-f9rl"
server_key: "726c31d58594dfb7565c037c203146e4"
server_key_iv: "aee58e5f441e57c52eb643972a92eff0"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:25.256274 30840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:04:25.278434 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:25.278707 30840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:04:25.278824 30840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "c43afefb05b84a6fa60decfdd03652be"
format_stamp: "Formatted at 2026-09-25 10:04:25 on dist-test-slave-f9rl"
server_key: "726c31d58594dfb7565c037c203146e4"
server_key_iv: "aee58e5f441e57c52eb643972a92eff0"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:25.278940 30840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:04:25.314837 30840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:04:25.318831 30840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:04:25.319039 30840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:04:25.328260 30840 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:45023
I20260925 10:04:25.328311 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:45023 every 8 connection(s)
I20260925 10:04:25.328797 30840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:04:25.329119 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 30840
I20260925 10:04:25.329273 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:04:25.329499 19643 external_mini_cluster.cc:1428] Setting key 58461bffafbef59d7c7629560a1b6cce
I20260925 10:04:25.329738 30908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:04:25 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:04:25.338421 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: Bootstrap starting.
I20260925 10:04:25.343288 30908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: Neither blocks nor log segments found. Creating new log.
I20260925 10:04:25.343779 30908 log.cc:824] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: Log is configured to *not* fsync() on all Append() calls
I20260925 10:04:25.345021 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: No bootstrap required, opened a new log
I20260925 10:04:25.346303 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:25.330330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.19.46.254:45023)
0925 10:04:25.330517 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:25.330530 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:25.330563 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:04:25.331032 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:25.331041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:25.331153 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:25.331345 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:04:25.333338 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:25.334551 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:25.335470 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:25.335975 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:25.336120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:25.340587 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:04:25.340593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:04:25.340593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:04:25.340608 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:04:25.343731 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:25.344218 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:25.344227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:25.344227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:25.344384 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:25.344941 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:25.344942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:25.344943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:25.345335 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:04:25.345551 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:25.345889 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 10:04:25.346046 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260925 10:04:25.347818 30908 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } }
I20260925 10:04:25.348031 30908 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:04:25.348088 30908 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43afefb05b84a6fa60decfdd03652be, State: Initialized, Role: FOLLOWER
I20260925 10:04:25.348284 30908 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } }
I20260925 10:04:25.348398 30908 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:04:25.348436 30908 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:04:25.348487 30908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:04:25.349241 30908 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } }
I20260925 10:04:25.349395 30908 leader_election.cc:302] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43afefb05b84a6fa60decfdd03652be; no voters:
I20260925 10:04:25.349689 30908 leader_election.cc:288] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:04:25.349881 30913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:04:25.350063 30908 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:04:25.350100 30913 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [term 1 LEADER]: Becoming Leader. State: Replica: c43afefb05b84a6fa60decfdd03652be, State: Running, Role: LEADER
I20260925 10:04:25.350258 30913 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } }
I20260925 10:04:25.351802 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43afefb05b84a6fa60decfdd03652be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } } }
I20260925 10:04:25.351908 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [sys.catalog]: This master's current role is: LEADER
I20260925 10:04:25.352144 30915 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43afefb05b84a6fa60decfdd03652be. Latest consensus state: current_term: 1 leader_uuid: "c43afefb05b84a6fa60decfdd03652be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43afefb05b84a6fa60decfdd03652be" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45023 } } }
I20260925 10:04:25.352334 30915 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be [sys.catalog]: This master's current role is: LEADER
I20260925 10:04:25.352516 30922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:04:25.353380 30922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:04:25.353463 30908 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:04:25.368798 30922 catalog_manager.cc:1347] Generated new cluster ID: e89f40052c954e0d9365c63b8c92d83a
I20260925 10:04:25.369138 30922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:04:25.414141 30922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:04:25.414944 30922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:04:25.419021 30922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: Generated new TSK 0
I20260925 10:04:25.419402 30922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:04:26.419968 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:04:26.526731 30953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:04:26.526997 30953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:04:26.527026 30953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:04:26.528502 30953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:04:26.528573 30953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:04:26.528609 30953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:04:26.530761 30953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.46.212:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:04:26.531113 30953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:04:26.531420 30953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:04:26.539510 30958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:04:26.539922 30953 server_base.cc:1034] running on GCE node
W20260925 10:04:26.539501 30961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:26.539824 30959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:04:26.540460 30953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:04:26.540906 30953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:04:26.542109 30953 hybrid_clock.cc:648] HybridClock initialized: now 1790330666542079 us; error 51 us; skew 500 ppm
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:04:26.545308 30953 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:04:26.545989 30953 webserver.cc:460] Webserver started at http://127.19.46.193:35813/ using document root <none> and password file <none>
I20260925 10:04:26.546298 30953 fs_manager.cc:359] Metadata directory not provided
I20260925 10:04:26.546377 30953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:04:26.546530 30953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:04:26.565770 30953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "9a12172e1b514fc19fabf5260c276612"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "fac9870728a828892c16acf9fb967b57"
server_key_iv: "4f11b2bcd0753ff71a240d9a1da836f3"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.566730 30953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "9a12172e1b514fc19fabf5260c276612"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "fac9870728a828892c16acf9fb967b57"
server_key_iv: "4f11b2bcd0753ff71a240d9a1da836f3"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.571132 30953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260925 10:04:26.592569 30968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.593199 30953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:04:26.593345 30953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "9a12172e1b514fc19fabf5260c276612"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "fac9870728a828892c16acf9fb967b57"
server_key_iv: "4f11b2bcd0753ff71a240d9a1da836f3"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.593567 30953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:04:26.624346 30953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:04:26.627611 30953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:04:26.627867 30953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:04:26.628192 30953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:04:26.628656 30953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:04:26.628708 30953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.628741 30953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:04:26.628759 30953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.637383 30953 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:43689
I20260925 10:04:26.637737 31083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:43689 every 8 connection(s)
I20260925 10:04:26.638161 30953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:04:26.640425 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 30953
I20260925 10:04:26.640551 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:04:26.640771 19643 external_mini_cluster.cc:1428] Setting key d0e3ad2d028202a3063c86d3d1bc517d
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:04:26.652627 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:26.639075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:45645 (local address 127.19.46.254:45023)
0925 10:04:26.640330 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:26.640333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:26.640344 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:04:26.640377 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:26.640379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:26.640395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:26.641501 (+ 1106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:04:26.641543 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.642400 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.643857 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.644070 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.644173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:26.647531 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:04:26.647534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:04:26.647534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:04:26.647545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:04:26.649806 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:26.650570 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:26.650572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:26.650573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:26.650649 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:26.651022 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:26.651024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:26.651024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:26.651199 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:04:26.651292 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:26.652286 (+ 994us) server_negotiation.cc:299] Negotiation successful
0925 10:04:26.652455 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":60,"threads_started":1}
I20260925 10:04:26.653412 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:26.639083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45023 (local address 127.19.46.193:45645)
0925 10:04:26.639347 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:26.639382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:26.640069 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:26.640599 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:26.640604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:26.640913 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:04:26.641381 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:26.641388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.642529 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.642530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:26.643757 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:26.643760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.644200 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.644201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:26.644384 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:26.645157 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:04:26.645171 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:04:26.647362 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:04:26.649944 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:26.649958 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:26.649962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:26.650460 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:26.650737 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:26.650738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:26.650739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:26.650911 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:26.651362 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:04:26.651371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:04:26.651968 (+ 597us) client_negotiation.cc:769] Sending connection context
0925 10:04:26.652872 (+ 904us) client_negotiation.cc:240] Negotiation successful
0925 10:04:26.653127 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260925 10:04:26.654487 31084 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45023
I20260925 10:04:26.654611 31084 heartbeater.cc:464] Registering TS with master...
I20260925 10:04:26.654855 31084 heartbeater.cc:511] Master 127.19.46.254:45023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:04:26.655610 30872 ts_manager.cc:194] Registered new tserver with Master: 9a12172e1b514fc19fabf5260c276612 (127.19.46.193:43689)
I20260925 10:04:26.657164 30872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:45645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:04:26.689301 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:04:26.848922 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:04:26.849510 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:04:26.849591 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:04:26.852928 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:04:26.853075 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:04:26.853147 31090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:04:26.858371 31090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.46.212:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:04:26.858960 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:04:26.859380 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:04:26.870406 31098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:26.871320 31100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:26.871578 31097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:04:26.873248 31090 server_base.cc:1034] running on GCE node
I20260925 10:04:26.873627 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:04:26.873965 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:04:26.875154 31090 hybrid_clock.cc:648] HybridClock initialized: now 1790330666875124 us; error 46 us; skew 500 ppm
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:04:26.880854 31090 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:04:26.882267 31090 webserver.cc:460] Webserver started at http://127.19.46.194:38135/ using document root <none> and password file <none>
I20260925 10:04:26.882637 31090 fs_manager.cc:359] Metadata directory not provided
I20260925 10:04:26.883046 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:04:26.883437 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:04:26 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:04:26.903285 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "158cfa29d59f4a4099e794df7192e110"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "f08c88a7fa85b22044d7801fd39e995c"
server_key_iv: "9dc426e9edcc5de4da01dae6a15a43e9"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.903905 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "158cfa29d59f4a4099e794df7192e110"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "f08c88a7fa85b22044d7801fd39e995c"
server_key_iv: "9dc426e9edcc5de4da01dae6a15a43e9"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.908334 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:04:26.940636 31107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.941217 31090 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260925 10:04:26.941509 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "158cfa29d59f4a4099e794df7192e110"
format_stamp: "Formatted at 2026-09-25 10:04:26 on dist-test-slave-f9rl"
server_key: "f08c88a7fa85b22044d7801fd39e995c"
server_key_iv: "9dc426e9edcc5de4da01dae6a15a43e9"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:26.941668 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:04:26.969359 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:04:26.972409 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:04:26.972692 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:04:26.973042 31090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:04:26.973536 31090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:04:26.973636 31090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.973731 31090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:04:26.973798 31090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:26.982590 31090 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37607
I20260925 10:04:26.983438 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:04:26.988303 31221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37607 every 8 connection(s)
I20260925 10:04:26.989709 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 31090
I20260925 10:04:26.989861 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:04:26.990108 19643 external_mini_cluster.cc:1428] Setting key daa6a28dd0af980a6efdaa35f9b4b376
Sep 25 10:04:27 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330666, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:04:27.014008 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:26.989180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45023 (local address 127.19.46.194:47883)
0925 10:04:26.991404 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:26.991476 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:26.992254 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:26.992941 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:26.992971 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:26.993586 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:04:26.993930 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:26.993940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.995041 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.995042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:26.996486 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:26.996490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.998077 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.998080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:26.998300 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:26.999626 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:04:26.999642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:04:27.006309 (+ 6667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:04:27.011323 (+ 5014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:27.011342 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:27.011345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:27.011831 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:27.012133 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:27.012134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:27.012135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:27.012581 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:27.012991 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:04:27.012998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:04:27.013330 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.013500 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.013713 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2154,"thread_start_us":681,"threads_started":1}
I20260925 10:04:27.014506 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:26.989209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:47883 (local address 127.19.46.254:45023)
0925 10:04:26.989357 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:26.989360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:26.991577 (+ 2217us) server_negotiation.cc:407] Connection header received
0925 10:04:26.992411 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:26.992413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:26.992432 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:26.992467 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:04:26.994080 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.994888 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.996627 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:26.997541 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:26.997613 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.006524 (+ 8911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:04:27.006527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:04:27.006527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:04:27.006538 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:04:27.010997 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.011968 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.011970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.011970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.012050 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.012692 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.012694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.012695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.012874 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:04:27.012951 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.014243 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.014365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 10:04:27.015234 31222 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45023
I20260925 10:04:27.015354 31222 heartbeater.cc:464] Registering TS with master...
I20260925 10:04:27.015568 31222 heartbeater.cc:511] Master 127.19.46.254:45023 requested a full tablet report, sending...
I20260925 10:04:27.015969 30872 ts_manager.cc:194] Registered new tserver with Master: 158cfa29d59f4a4099e794df7192e110 (127.19.46.194:37607)
I20260925 10:04:27.016721 30872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:47883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:04:27.054258 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:04:27.214829 31228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:04:27.215283 31228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:04:27.215399 31228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:04:27.217857 31228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:04:27.218212 31228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:04:27.218343 31228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:04:27.221195 31228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.46.212:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:04:27.221865 31228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:04:27.222275 31228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:04:27.230926 31234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:04:27.231419 31233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:04:27.231662 31228 server_base.cc:1034] running on GCE node
W20260925 10:04:27.232007 31236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:04:27.232373 31228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:04:27.232771 31228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:04:27.233969 31228 hybrid_clock.cc:648] HybridClock initialized: now 1790330667233925 us; error 57 us; skew 500 ppm
Sep 25 10:04:27 dist-test-slave-f9rl krb5kdc[30392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330667, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:04:27.238140 31228 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:04:27.238909 31228 webserver.cc:460] Webserver started at http://127.19.46.195:38183/ using document root <none> and password file <none>
I20260925 10:04:27.239435 31228 fs_manager.cc:359] Metadata directory not provided
I20260925 10:04:27.239583 31228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:04:27.239820 31228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:04:27 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330667, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:04:27.259498 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "236b9ac82f8742c597613fc4eabd8f40"
format_stamp: "Formatted at 2026-09-25 10:04:27 on dist-test-slave-f9rl"
server_key: "64cd0612c079b890b49842f5800c94cc"
server_key_iv: "3c8ab23af0223b638ae9cb807c4b6a3e"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:27.259974 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "236b9ac82f8742c597613fc4eabd8f40"
format_stamp: "Formatted at 2026-09-25 10:04:27 on dist-test-slave-f9rl"
server_key: "64cd0612c079b890b49842f5800c94cc"
server_key_iv: "3c8ab23af0223b638ae9cb807c4b6a3e"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:27.263907 31228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 10:04:27.283272 31243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:27.283593 31228 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260925 10:04:27.283707 31228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "236b9ac82f8742c597613fc4eabd8f40"
format_stamp: "Formatted at 2026-09-25 10:04:27 on dist-test-slave-f9rl"
server_key: "64cd0612c079b890b49842f5800c94cc"
server_key_iv: "3c8ab23af0223b638ae9cb807c4b6a3e"
server_key_version: "kuduclusterkey@0"
I20260925 10:04:27.283828 31228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:04:27.321549 31228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:04:27.327698 31228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:04:27.327896 31228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:04:27.328130 31228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:04:27.328701 31228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:04:27.328833 31228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:27.328873 31228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:04:27.328892 31228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:04:27.337659 31228 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:45833
I20260925 10:04:27.339056 31228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:04:27.339504 31357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:45833 every 8 connection(s)
I20260925 10:04:27.343281 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 31228
I20260925 10:04:27.343405 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:04:27.343650 19643 external_mini_cluster.cc:1428] Setting key 4ee72c38ea5392ba9eb268dfaa26bee6
Sep 25 10:04:27 dist-test-slave-f9rl krb5kdc[30392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330667, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:04:27.362682 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.341945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:46245 (local address 127.19.46.254:45023)
0925 10:04:27.342009 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.342033 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.344165 (+ 2132us) server_negotiation.cc:407] Connection header received
0925 10:04:27.345468 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.345471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.345488 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.345525 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:04:27.348789 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.350188 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.352437 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.352795 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.352882 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.356275 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:04:27.356277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:04:27.356277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:04:27.356288 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:04:27.359060 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.359965 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.359967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.359968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.360082 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.361221 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.361223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.361223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.361572 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:04:27.361672 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.362406 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.362543 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:04:27.362994 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.341928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45023 (local address 127.19.46.195:46245)
0925 10:04:27.343645 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.343837 (+ 192us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.345185 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.346610 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.346618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.347509 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:04:27.348509 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.348532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.350349 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.350359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.352253 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.352259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.352916 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.352917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.353117 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.353961 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:04:27.353972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:04:27.356085 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:04:27.359405 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:27.359425 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:27.359429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:27.359848 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:27.360218 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:04:27.360219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:04:27.360220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:04:27.360597 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:04:27.361735 (+ 1138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:04:27.361745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:04:27.362168 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.362343 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.362708 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"thread_start_us":131,"threads_started":1}
I20260925 10:04:27.364615 31358 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45023
I20260925 10:04:27.364744 31358 heartbeater.cc:464] Registering TS with master...
I20260925 10:04:27.364966 31358 heartbeater.cc:511] Master 127.19.46.254:45023 requested a full tablet report, sending...
I20260925 10:04:27.365549 30872 ts_manager.cc:194] Registered new tserver with Master: 236b9ac82f8742c597613fc4eabd8f40 (127.19.46.195:45833)
I20260925 10:04:27.366218 30872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:46245
I20260925 10:04:27.369235 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:04:27.397886 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.381224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.19.46.254:45023)
0925 10:04:27.385554 (+ 4330us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.385558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.388777 (+ 3219us) server_negotiation.cc:407] Connection header received
0925 10:04:27.388917 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.388919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.388934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.388969 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:04:27.390170 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.390933 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.391807 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.392042 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.392143 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.394233 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:04:27.394235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:04:27.394235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:04:27.394245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:04:27.396357 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.396866 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.396867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.396868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.396935 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:04:27.397150 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:04:27.397151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:04:27.397152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:04:27.397313 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:04:27.397408 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.397678 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.397773 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4283}
I20260925 10:04:27.400406 30872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:04:27.403810 30872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:04:27.419035 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.408839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43689 (local address 127.0.0.1:57636)
0925 10:04:27.410578 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.410586 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.410619 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.411028 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.411031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.411034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:04:27.411107 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.411109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.412822 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.412824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.417560 (+ 4736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.417563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.418674 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.418675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.418777 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.418794 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.418873 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.418940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":129,"threads_started":1}
I20260925 10:04:27.421754 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.409534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.19.46.193:43689)
0925 10:04:27.410713 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.410718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.410729 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:04:27.410765 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.410768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.410785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.410860 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:04:27.411190 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.412723 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.417661 (+ 4938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.418484 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.418563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.418633 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.421569 (+ 2936us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.421649 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":61,"threads_started":1}
I20260925 10:04:27.424255 31018 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc704bdae354eabaeafb54aed69c775 (DEFAULT_TABLE table=test-table [id=60857ac5fe46457c87c002994211d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:04:27.424700 31018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc704bdae354eabaeafb54aed69c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:04:27.431452 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.408495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37607 (local address 127.0.0.1:47086)
0925 10:04:27.410353 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.410407 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.410477 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.421194 (+ 10717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.421195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.421199 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:04:27.421251 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.421254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.428326 (+ 7072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.428327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.429556 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.429560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.431055 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.431056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.431130 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.431136 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.431193 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.431330 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1823,"thread_start_us":166,"threads_started":1}
I20260925 10:04:27.431697 31377 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: Bootstrap starting.
I20260925 10:04:27.432103 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.409835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.19.46.195:45833)
0925 10:04:27.410980 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.410984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.414735 (+ 3751us) server_negotiation.cc:407] Connection header received
0925 10:04:27.414787 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.414790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.414807 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.414879 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:04:27.415091 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.416421 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.430240 (+ 13819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.431361 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.431432 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.431523 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.431982 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.432026 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":119,"threads_started":1}
I20260925 10:04:27.432086 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.409084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:45833 (local address 127.0.0.1:53360)
0925 10:04:27.414609 (+ 5525us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.414621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.414669 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.414970 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.414971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.414973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:04:27.415020 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.415021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.416537 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.416538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.427931 (+ 11393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.427936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.431810 (+ 3874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.431811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.431882 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.431887 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.431961 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.432013 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5504,"thread_start_us":152,"threads_started":1}
I20260925 10:04:27.432490 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.408611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.19.46.194:37607)
0925 10:04:27.420885 (+ 12274us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.420892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.420910 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:04:27.420961 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.420964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.420985 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.421082 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:04:27.422086 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.423345 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.429716 (+ 6371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.430604 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.430683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.431005 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.432333 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.432397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12218,"thread_start_us":104,"threads_started":1}
I20260925 10:04:27.433598 31155 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc704bdae354eabaeafb54aed69c775 (DEFAULT_TABLE table=test-table [id=60857ac5fe46457c87c002994211d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:04:27.433596 31292 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc704bdae354eabaeafb54aed69c775 (DEFAULT_TABLE table=test-table [id=60857ac5fe46457c87c002994211d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:04:27.433990 31155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc704bdae354eabaeafb54aed69c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:04:27.433990 31292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc704bdae354eabaeafb54aed69c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:04:27.437841 31379 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Bootstrap starting.
I20260925 10:04:27.439015 31379 tablet_bootstrap.cc:654] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Neither blocks nor log segments found. Creating new log.
I20260925 10:04:27.439131 31377 tablet_bootstrap.cc:654] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: Neither blocks nor log segments found. Creating new log.
I20260925 10:04:27.439476 31377 log.cc:824] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: Log is configured to *not* fsync() on all Append() calls
I20260925 10:04:27.439464 31379 log.cc:824] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Log is configured to *not* fsync() on all Append() calls
I20260925 10:04:27.440832 31378 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Bootstrap starting.
I20260925 10:04:27.442485 31378 tablet_bootstrap.cc:654] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Neither blocks nor log segments found. Creating new log.
I20260925 10:04:27.443061 31379 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: No bootstrap required, opened a new log
I20260925 10:04:27.443061 31377 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: No bootstrap required, opened a new log
I20260925 10:04:27.443172 31377 ts_tablet_manager.cc:1397] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 10:04:27.443336 31379 ts_tablet_manager.cc:1397] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 10:04:27.446570 31379 raft_consensus.cc:348] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.447539 31378 log.cc:824] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Log is configured to *not* fsync() on all Append() calls
I20260925 10:04:27.447677 31377 raft_consensus.cc:348] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.447865 31377 raft_consensus.cc:374] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:04:27.447907 31377 raft_consensus.cc:729] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a12172e1b514fc19fabf5260c276612, State: Initialized, Role: FOLLOWER
I20260925 10:04:27.448232 31379 raft_consensus.cc:374] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:04:27.448393 31379 raft_consensus.cc:729] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236b9ac82f8742c597613fc4eabd8f40, State: Initialized, Role: FOLLOWER
I20260925 10:04:27.448212 31377 consensus_queue.cc:260] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.448673 31377 ts_tablet_manager.cc:1428] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:04:27.448805 31084 heartbeater.cc:503] Master 127.19.46.254:45023 was elected leader, sending a full tablet report...
I20260925 10:04:27.448666 31379 consensus_queue.cc:260] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.449172 31379 ts_tablet_manager.cc:1428] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 10:04:27.449334 31358 heartbeater.cc:503] Master 127.19.46.254:45023 was elected leader, sending a full tablet report...
I20260925 10:04:27.450476 31378 tablet_bootstrap.cc:492] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: No bootstrap required, opened a new log
I20260925 10:04:27.450629 31378 ts_tablet_manager.cc:1397] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260925 10:04:27.452661 31378 raft_consensus.cc:348] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.452877 31378 raft_consensus.cc:374] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:04:27.452914 31378 raft_consensus.cc:729] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158cfa29d59f4a4099e794df7192e110, State: Initialized, Role: FOLLOWER
I20260925 10:04:27.453065 31378 consensus_queue.cc:260] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.453359 31378 ts_tablet_manager.cc:1428] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:04:27.453495 31222 heartbeater.cc:503] Master 127.19.46.254:45023 was elected leader, sending a full tablet report...
I20260925 10:04:27.462980 31384 raft_consensus.cc:482] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:04:27.463140 31384 raft_consensus.cc:504] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.463565 31384 leader_election.cc:288] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158cfa29d59f4a4099e794df7192e110 (127.19.46.194:37607), 9a12172e1b514fc19fabf5260c276612 (127.19.46.193:43689)
I20260925 10:04:27.468168 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.463911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37607 (local address 127.19.46.195:43885)
0925 10:04:27.464198 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.464211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.464269 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.464491 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.464492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.464494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:04:27.464574 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.464576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.465682 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.465683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.466998 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.467002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.467860 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.467861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.467951 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.467956 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.468007 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.468056 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260925 10:04:27.468750 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.463981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:43885 (local address 127.19.46.194:37607)
0925 10:04:27.464033 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.464036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.464254 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:04:27.464332 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.464333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.464346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.464431 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:04:27.464675 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.465526 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.467127 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.467733 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.467812 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.467822 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.468616 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.468680 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:04:27.469058 31175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc704bdae354eabaeafb54aed69c775" candidate_uuid: "236b9ac82f8742c597613fc4eabd8f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158cfa29d59f4a4099e794df7192e110" is_pre_election: true
I20260925 10:04:27.469233 31175 raft_consensus.cc:2454] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 236b9ac82f8742c597613fc4eabd8f40 in term 0.
I20260925 10:04:27.469501 31244 leader_election.cc:302] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158cfa29d59f4a4099e794df7192e110, 236b9ac82f8742c597613fc4eabd8f40; no voters:
I20260925 10:04:27.469729 31384 raft_consensus.cc:2781] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:04:27.469790 31384 raft_consensus.cc:482] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:04:27.469821 31384 raft_consensus.cc:3037] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:04:27.470817 31384 raft_consensus.cc:504] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.470978 31384 leader_election.cc:288] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [CANDIDATE]: Term 1 election: Requested vote from peers 158cfa29d59f4a4099e794df7192e110 (127.19.46.194:37607), 9a12172e1b514fc19fabf5260c276612 (127.19.46.193:43689)
I20260925 10:04:27.471225 31175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc704bdae354eabaeafb54aed69c775" candidate_uuid: "236b9ac82f8742c597613fc4eabd8f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158cfa29d59f4a4099e794df7192e110"
I20260925 10:04:27.471328 31175 raft_consensus.cc:3037] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:04:27.472252 31175 raft_consensus.cc:2454] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 236b9ac82f8742c597613fc4eabd8f40 in term 1.
I20260925 10:04:27.472448 31244 leader_election.cc:302] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158cfa29d59f4a4099e794df7192e110, 236b9ac82f8742c597613fc4eabd8f40; no voters:
I20260925 10:04:27.472546 31384 raft_consensus.cc:2781] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:04:27.472702 31384 raft_consensus.cc:686] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 1 LEADER]: Becoming Leader. State: Replica: 236b9ac82f8742c597613fc4eabd8f40, State: Running, Role: LEADER
I20260925 10:04:27.472808 31384 consensus_queue.cc:237] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } }
I20260925 10:04:27.473752 30872 catalog_manager.cc:5495] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 236b9ac82f8742c597613fc4eabd8f40 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "236b9ac82f8742c597613fc4eabd8f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "236b9ac82f8742c597613fc4eabd8f40" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45833 } health_report { overall_health: HEALTHY } } }
I20260925 10:04:27.478422 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.464032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43689 (local address 127.19.46.195:51227)
0925 10:04:27.468276 (+ 4244us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.468289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:04:27.468339 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:04:27.474830 (+ 6491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:04:27.474832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:04:27.474834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:04:27.474889 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.474890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.475942 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.475944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.477213 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:04:27.477216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.478151 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.478153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:04:27.478223 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.478227 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:04:27.478279 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:04:27.478328 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4231,"thread_start_us":121,"threads_started":1}
I20260925 10:04:27.478772 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.464409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:51227 (local address 127.19.46.193:43689)
0925 10:04:27.464454 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.464456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.468469 (+ 4013us) server_negotiation.cc:407] Connection header received
0925 10:04:27.468502 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.468503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.468519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.468565 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:04:27.474988 (+ 6423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.475831 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.477336 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.478020 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.478101 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.478112 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.478634 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.478701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:04:27.479090 31038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc704bdae354eabaeafb54aed69c775" candidate_uuid: "236b9ac82f8742c597613fc4eabd8f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a12172e1b514fc19fabf5260c276612" is_pre_election: true
I20260925 10:04:27.479271 31038 raft_consensus.cc:2454] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 236b9ac82f8742c597613fc4eabd8f40 in term 0.
I20260925 10:04:27.479390 31037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc704bdae354eabaeafb54aed69c775" candidate_uuid: "236b9ac82f8742c597613fc4eabd8f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a12172e1b514fc19fabf5260c276612"
I20260925 10:04:27.479441 31037 raft_consensus.cc:3037] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:04:27.480464 31037 raft_consensus.cc:2454] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 236b9ac82f8742c597613fc4eabd8f40 in term 1.
W20260925 10:04:27.489478 31223 tablet.cc:2369] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:04:27.491009 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.486335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.19.46.195:45833)
0925 10:04:27.486379 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:04:27.486381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:04:27.486388 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:04:27.486429 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:04:27.486430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:04:27.486451 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:04:27.486515 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:04:27.486785 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.487583 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.488833 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:04:27.489156 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:04:27.489234 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:04:27.490292 (+ 1058us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:04:27.490441 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:04:27.490519 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:04:27.490869 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 10:04:27.490899 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:04:27.493511 31037 raft_consensus.cc:1261] T 4cc704bdae354eabaeafb54aed69c775 P 9a12172e1b514fc19fabf5260c276612 [term 1 FOLLOWER]: Refusing update from remote peer 236b9ac82f8742c597613fc4eabd8f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:04:27.493973 31387 consensus_queue.cc:1035] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a12172e1b514fc19fabf5260c276612" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:04:27.496423 31175 raft_consensus.cc:1261] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 1 FOLLOWER]: Refusing update from remote peer 236b9ac82f8742c597613fc4eabd8f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:04:27.496870 31387 consensus_queue.cc:1035] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cfa29d59f4a4099e794df7192e110" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:04:27.504406 31390 mvcc.cc:205] Tried to move back new op lower bound from 7333194414049996800 to 7333194413969264640. Current Snapshot: MvccSnapshot[applied={T|T < 7333194414049996800}]
I20260925 10:04:27.505992 31393 mvcc.cc:205] Tried to move back new op lower bound from 7333194414049996800 to 7333194413969264640. Current Snapshot: MvccSnapshot[applied={T|T < 7333194414049996800}]
I20260925 10:04:27.509851 31391 mvcc.cc:205] Tried to move back new op lower bound from 7333194414049996800 to 7333194413969264640. Current Snapshot: MvccSnapshot[applied={T|T < 7333194414049996800}]
I20260925 10:04:27.513065 30872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:04:27.513195 30872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:04:27.514192 30872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43afefb05b84a6fa60decfdd03652be: Sending DeleteTablet for 3 replicas of tablet 4cc704bdae354eabaeafb54aed69c775
I20260925 10:04:27.514943 31155 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc704bdae354eabaeafb54aed69c775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:04:27 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:47086
I20260925 10:04:27.515192 31018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc704bdae354eabaeafb54aed69c775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:04:27 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:57636
I20260925 10:04:27.515652 31292 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc704bdae354eabaeafb54aed69c775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:04:27 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:53360
I20260925 10:04:27.516574 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 30953
I20260925 10:04:27.521852 31399 tablet_replica.cc:330] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: stopping tablet replica
I20260925 10:04:27.522346 31399 raft_consensus.cc:2229] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:04:27.522594 31399 raft_consensus.cc:2258] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:04:27.528908 31401 tablet_replica.cc:330] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: stopping tablet replica
I20260925 10:04:27.529134 31401 raft_consensus.cc:2229] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:04:27.529431 31401 raft_consensus.cc:2258] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:04:27.530030 31399 ts_tablet_manager.cc:1905] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:04:27.537881 31401 ts_tablet_manager.cc:1905] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:04:27.540405 31399 ts_tablet_manager.cc:1918] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:04:27.540551 31399 log.cc:1196] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/4cc704bdae354eabaeafb54aed69c775
I20260925 10:04:27.541342 31399 ts_tablet_manager.cc:1939] T 4cc704bdae354eabaeafb54aed69c775 P 158cfa29d59f4a4099e794df7192e110: Deleting consensus metadata
I20260925 10:04:27.541706 31401 ts_tablet_manager.cc:1918] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:04:27.541798 31401 log.cc:1196] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/4cc704bdae354eabaeafb54aed69c775
I20260925 10:04:27.542080 31401 ts_tablet_manager.cc:1939] T 4cc704bdae354eabaeafb54aed69c775 P 236b9ac82f8742c597613fc4eabd8f40: Deleting consensus metadata
I20260925 10:04:27.542640 30859 catalog_manager.cc:4841] TS 236b9ac82f8742c597613fc4eabd8f40 (127.19.46.195:45833): tablet 4cc704bdae354eabaeafb54aed69c775 (table test-table [id=60857ac5fe46457c87c002994211d183]) successfully deleted
I20260925 10:04:27.543238 30856 catalog_manager.cc:4841] TS 158cfa29d59f4a4099e794df7192e110 (127.19.46.194:37607): tablet 4cc704bdae354eabaeafb54aed69c775 (table test-table [id=60857ac5fe46457c87c002994211d183]) successfully deleted
W20260925 10:04:27.543542 30856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:43689 (error 108)
I20260925 10:04:27.545584 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 31090
I20260925 10:04:27.550068 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:04:27.549910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:43689 (local address 127.0.0.1:57648)
0925 10:04:27.549964 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:04:27.550017 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:04:27.550266 30856 catalog_manager.cc:4584] TS 9a12172e1b514fc19fabf5260c276612 (127.19.46.193:43689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cc704bdae354eabaeafb54aed69c775: Network error: Client connection negotiation failed: client connection to 127.19.46.193:43689: connect: Connection refused (error 111)
I20260925 10:04:27.554940 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 31228
I20260925 10:04:27.562600 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 30840
I20260925 10:04:27.569118 19643 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:04:27.916069 19643 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:04:27.916304 19643 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:04:28.262722 19643 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:04:28.263 UTC [30422] LOG: received smart shutdown request
2026-09-25 10:04:28.269 UTC [30422] LOG: background worker "logical replication launcher" (PID 30431) exited with exit code 1
2026-09-25 10:04:28.270 UTC [30426] LOG: shutting down
2026-09-25 10:04:28.598 UTC [30422] LOG: database system is shut down
2026-09-25T10:04:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54928 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:04:28 dist-test-slave-f9rl krb5kdc[31411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:04:28 dist-test-slave-f9rl krb5kdc[31411](info): set up 2 sockets
Sep 25 10:04:28 dist-test-slave-f9rl krb5kdc[31411](info): commencing operation
krb5kdc: starting...
W20260925 10:04:30.071473 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:04:30 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:04:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangeradmin_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangeradmin_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangerlookup_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangerlookup_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
I20260925 10:04:30.318984 19643 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task46RfQ8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/postgres -l logfile start
2026-09-25 10:04:34.248 UTC [31442] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:04:34.249 UTC [31442] LOG: listening on IPv4 address "127.19.46.212", port 42713
2026-09-25 10:04:34.256 UTC [31442] LOG: listening on Unix socket "/tmp/.s.PGSQL.42713"
2026-09-25 10:04:34.287 UTC [31445] LOG: database system was shut down at 2026-09-25 10:04:31 UTC
2026-09-25 10:04:34.293 UTC [31442] LOG: database system is ready to accept connections
I20260925 10:04:35.416105 19643 mini_postgres.cc:96] Postgres bound to 42713
2026-09-25 10:04:35.425 UTC [31453] FATAL: database "slave" does not exist
127.19.46.212:42713 - accepting connections
I20260925 10:04:35.426417 19643 mini_ranger.cc:161] Starting Ranger...
I20260925 10:04:35.448972 19643 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:04:36.273342 19643 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:04:36.273425 19643 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin
2026-09-25 10:04:36,590 [I] DB FLAVOR :POSTGRES
2026-09-25 10:04:36,591 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:04:36,591 [I] Checking connection..
2026-09-25 10:04:36,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:04:36,936 [I] Checking connection passed.
2026-09-25 10:04:36,936 [I] --------- Verifying version history table ---------
2026-09-25 10:04:36,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:04:37,269 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:04:37,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:04:37,634 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:04:37,634 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:04:37,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:04:37.970 UTC [31517] WARNING: there is no transaction in progress
2026-09-25 10:04:37,978 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:04:37,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:04:38,332 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:04:38,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:04:38,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:04:38,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:04:39,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rl.c.gcp-upstream.internal','N') ;"
2026-09-25 10:04:39,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:04:39,723 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:04:39,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:04:40,056 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:04:40,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:04:40,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:04:40,658 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:04:40,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:04:42.200 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.221 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.245 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.268 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.290 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.314 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.394 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.403 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.422 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.440 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.460 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.480 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.494 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.505 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.518 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:42.529 UTC [31626] WARNING: there is no transaction in progress
2026-09-25 10:04:43,643 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:04:43,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:04:43,938 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:04:43,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:04:44,301 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:04:44,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:04:44,613 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:04:44,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:04:44,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:04:45,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rl.c.gcp-upstream.internal';"
2026-09-25 10:04:45,610 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:04:45,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:04:45,965 [I] DB_PATCHES have already been applied
I20260925 10:04:45.973194 19643 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task46RfQ8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:04:45.973320 19643 mini_ranger.cc:193] Using host: 127.19.46.212
I20260925 10:04:45.977629 19643 mini_ranger.cc:227] Ranger admin URL: http://127.19.46.212:34297
Sep 25, 2026 10:04:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:04:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:04:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:04:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:04:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:04:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:04:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:04:47.289171 19643 mini_ranger.cc:160] Time spent starting Ranger: real 11.863s user 0.003s sys 0.010s
Sep 25, 2026 10:05:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:05:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:05:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:05:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:05:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:05:19.405545 19643 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangerkms_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/rangerkms_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/HTTP_127.19.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:05:19 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330719, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:05:19.543715 19643 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:05:19.557005 19643 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:05:20.841603 19643 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:05:20.841714 19643 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms
I20260925 10:05:20.841735 19643 mini_ranger_kms.cc:227] Using postgres at 127.19.46.212:42713
2026-09-25 10:05:21,057 [I] DB FLAVOR :POSTGRES
2026-09-25 10:05:21,058 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:05:21,058 [I] Checking connection
2026-09-25 10:05:21,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:05:21,378 [I] connection success
2026-09-25 10:05:21,378 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:05:21,378 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:05:21,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:05:21,740 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:05:21,740 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:05:21,740 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:05:21,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.46.212:42713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:05:22,092 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:05:22.687203 19643 mini_ranger_kms.cc:310] Created kms service
I20260925 10:05:22.951581 19643 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:05:23.016247 19643 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:05:23 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330719, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:23.570120 19643 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:05:23.570343 19643 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms:/tmp/dist-test-task46RfQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task46RfQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:05:23.570369 19643 mini_ranger_kms.cc:240] Using host: 127.19.46.212
I20260925 10:05:23.574153 19643 mini_ranger_kms.cc:276] Ranger KMS PID: 31808
I20260925 10:05:23.574246 19643 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.46.212:39889
Sep 25, 2026 10:05:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:05:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:05:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:05:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:05:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:05:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 10:05:26 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330726, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:26.177385 19643 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.634s user 0.000s sys 0.199s
I20260925 10:05:26.177534 19643 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:05:26.177587 19643 mini_ranger_kms.cc:401] 127.19.46.212:39889/kms/v1/keys
Sep 25 10:05:26 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330726, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
Sep 25 10:05:27 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330726, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.46.212@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
Sep 25 10:05:30 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:05:30.075309 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45063
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:45063
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:30.165778 31858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:30.166038 31858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:30.166079 31858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:30.167470 31858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:30.167550 31858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:30.167577 31858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:30.167608 31858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:30.167632 31858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:30.169584 31858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.46.212:39889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:45063
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:30.169888 31858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:30.170156 31858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:30.175735 31864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:30.175902 31866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:30.175809 31858 server_base.cc:1034] running on GCE node
W20260925 10:05:30.175792 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:30.176370 31858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:30.176708 31858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:30.177877 31858 hybrid_clock.cc:648] HybridClock initialized: now 1790330730177874 us; error 50 us; skew 500 ppm
Sep 25 10:05:30 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330730, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:30.180611 31858 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:30.181114 31858 webserver.cc:460] Webserver started at http://127.19.46.254:38299/ using document root <none> and password file <none>
I20260925 10:05:30.181311 31858 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:30.181345 31858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:30.181499 31858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:05:30 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330730, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:30.233547 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "a9c2066058804c5d8b9d1ac44451b9eb"
format_stamp: "Formatted at 2026-09-25 10:05:30 on dist-test-slave-f9rl"
server_key: "81613a7a5b148b2b103008dc681ecd9f"
server_key_iv: "5eff033f618eb49011f7593d6f939667"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:30.234082 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "a9c2066058804c5d8b9d1ac44451b9eb"
format_stamp: "Formatted at 2026-09-25 10:05:30 on dist-test-slave-f9rl"
server_key: "81613a7a5b148b2b103008dc681ecd9f"
server_key_iv: "5eff033f618eb49011f7593d6f939667"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:30.236398 31858 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:05:30.255398 31873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:30.255635 31858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:05:30.255730 31858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "a9c2066058804c5d8b9d1ac44451b9eb"
format_stamp: "Formatted at 2026-09-25 10:05:30 on dist-test-slave-f9rl"
server_key: "81613a7a5b148b2b103008dc681ecd9f"
server_key_iv: "5eff033f618eb49011f7593d6f939667"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:30.255856 31858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:30.272548 31858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:30.275672 31858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:30.275949 31858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:30.282384 31858 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:45063
I20260925 10:05:30.282446 31926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:45063 every 8 connection(s)
I20260925 10:05:30.282910 31858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:30.283753 31927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:30.286405 31927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Bootstrap starting.
I20260925 10:05:30.287516 31927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:30.287979 31927 log.cc:824] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:30.289172 31927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: No bootstrap required, opened a new log
I20260925 10:05:30.291822 31927 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:30.292063 31927 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:30.292121 31927 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c2066058804c5d8b9d1ac44451b9eb, State: Initialized, Role: FOLLOWER
I20260925 10:05:30.292333 31927 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:30.292433 31927 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:30.292486 31927 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:30.292579 31927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:30.292740 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 31858
I20260925 10:05:30.292865 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:05:30.293073 19643 external_mini_cluster.cc:1428] Setting key ab4b1050713ea1013a1a22f64234e7b5
I20260925 10:05:30.293707 31927 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:30.293934 31927 leader_election.cc:302] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c2066058804c5d8b9d1ac44451b9eb; no voters:
I20260925 10:05:30.294170 31927 leader_election.cc:288] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:05:30.294198 31932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:30.294351 31932 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 LEADER]: Becoming Leader. State: Replica: a9c2066058804c5d8b9d1ac44451b9eb, State: Running, Role: LEADER
I20260925 10:05:30.294513 31932 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:30.294550 31927 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:30.295666 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } } }
I20260925 10:05:30.296007 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:30.295935 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9c2066058804c5d8b9d1ac44451b9eb. Latest consensus state: current_term: 1 leader_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } } }
I20260925 10:05:30.296304 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:30.296470 31943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:30.296845 31927 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:05:30.297068 31943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:05:30 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:30.317375 31943 catalog_manager.cc:1347] Generated new cluster ID: 2d352319743f4f8e977e70ca1544f4dc
I20260925 10:05:30.317487 31943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:05:30.319811 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:30.293834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.19.46.254:45063)
0925 10:05:30.294089 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:30.294102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:30.294799 (+ 697us) server_negotiation.cc:407] Connection header received
0925 10:05:30.295031 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:30.295040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:30.295106 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:30.295262 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:30.296368 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:30.304184 (+ 7816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:30.306036 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:30.307364 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:30.307581 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:30.314650 (+ 7069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:30.314658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:30.314658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:30.314678 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:30.317238 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:30.317840 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:30.317845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:30.317845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:30.317975 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:30.318262 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:30.318264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:30.318264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:30.318661 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:30.318847 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:30.319373 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 10:05:30.319523 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":113,"threads_started":1}
I20260925 10:05:30.338465 31943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:05:30.339130 31943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:30.350664 31943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Generated new TSK 0
I20260925 10:05:30.350901 31943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:05:31.230320 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:31.335781 31972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:31.336088 31972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:31.336139 31972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:31.337677 31972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:31.337759 31972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:31.337828 31972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:05:31.340597 31972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:31.340973 31972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:31.341670 31972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:31.347488 31977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.347502 31980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.347864 31978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:31.348570 31972 server_base.cc:1034] running on GCE node
I20260925 10:05:31.348878 31972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:31.349210 31972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:31.350409 31972 hybrid_clock.cc:648] HybridClock initialized: now 1790330731350382 us; error 34 us; skew 500 ppm
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:31.353974 31972 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:05:31.354606 31972 webserver.cc:460] Webserver started at http://127.19.46.193:34013/ using document root <none> and password file <none>
I20260925 10:05:31.354923 31972 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:31.355015 31972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:31.355185 31972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:31.375572 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "975b82f728c94884bdf0d71272e9d40e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "c447ad10b4f45bfc71a65d352d06d2e3"
server_key_iv: "19e380793a9c9d60bf51ab0281461bfc"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.376121 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "975b82f728c94884bdf0d71272e9d40e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "c447ad10b4f45bfc71a65d352d06d2e3"
server_key_iv: "19e380793a9c9d60bf51ab0281461bfc"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.379436 31972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:05:31.394114 31987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.394433 31972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:31.394557 31972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "975b82f728c94884bdf0d71272e9d40e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "c447ad10b4f45bfc71a65d352d06d2e3"
server_key_iv: "19e380793a9c9d60bf51ab0281461bfc"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.394699 31972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:31.414546 31972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:31.416894 31972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:31.417075 31972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:31.417361 31972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:31.417802 31972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:31.417858 31972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.417910 31972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:31.417937 31972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.426476 31972 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:33415
I20260925 10:05:31.427358 31972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:05:31.428009 32101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:33415 every 8 connection(s)
I20260925 10:05:31.429962 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 31972
I20260925 10:05:31.430087 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:05:31.430269 19643 external_mini_cluster.cc:1428] Setting key ee6d873a9ede71d65b8c771f072cf8c9
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:31.445854 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.428684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56993 (local address 127.19.46.254:45063)
0925 10:05:31.429764 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:31.429767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:31.430476 (+ 709us) server_negotiation.cc:407] Connection header received
0925 10:05:31.430509 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:31.430510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:31.430527 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:31.430591 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:31.431593 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.432916 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.435415 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.435814 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.435865 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.440838 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:31.440841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:31.440842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:31.440853 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:31.443531 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.444363 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.444365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.444366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.444452 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.444808 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.444810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.444810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.444995 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:31.445062 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:31.445589 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 10:05:31.445695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:05:31.447052 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.428310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.193:56993)
0925 10:05:31.428832 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:31.428880 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:31.429633 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:31.430703 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:31.430709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:31.431160 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:31.431480 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.431488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.433102 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.433103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.435235 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.435241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.435967 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.435968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.436080 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.438723 (+ 2643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:31.438739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:31.440670 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:31.443703 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.443719 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.443723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.444158 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.444540 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.444541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.444541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.444715 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.445104 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:31.445111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:31.445336 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 10:05:31.446470 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0925 10:05:31.446688 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":104,"threads_started":1}
I20260925 10:05:31.448009 32103 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:31.448150 32103 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:31.448408 32103 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:31.448963 31890 ts_manager.cc:194] Registered new tserver with Master: 975b82f728c94884bdf0d71272e9d40e (127.19.46.193:33415)
I20260925 10:05:31.449905 31890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:56993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:05:31.473930 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:31.590951 32110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:31.591221 32110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:31.591349 32110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:31.592718 32110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:31.592803 32110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:31.592852 32110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:05:31.594693 32110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:31.595006 32110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:31.595285 32110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:31.601840 32120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.601915 32118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.602516 32117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:31.602485 32110 server_base.cc:1034] running on GCE node
I20260925 10:05:31.602809 32110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:31.603094 32110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:31.606465 32110 hybrid_clock.cc:648] HybridClock initialized: now 1790330731606421 us; error 54 us; skew 500 ppm
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:31.609881 32110 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:05:31.610396 32110 webserver.cc:460] Webserver started at http://127.19.46.194:36425/ using document root <none> and password file <none>
I20260925 10:05:31.610731 32110 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:31.610992 32110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:31.611218 32110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:31.628392 32110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "ee01d250c30f401c9adeb2e8ae65936e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "0b2d48686c9677fd0c14a2e49294b092"
server_key_iv: "961b4d0e12702d3eb5cbc4c8a6f83b49"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.628927 32110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "ee01d250c30f401c9adeb2e8ae65936e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "0b2d48686c9677fd0c14a2e49294b092"
server_key_iv: "961b4d0e12702d3eb5cbc4c8a6f83b49"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.642037 32110 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.000s
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:31.663097 32127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.663457 32110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 10:05:31.663566 32110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "ee01d250c30f401c9adeb2e8ae65936e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "0b2d48686c9677fd0c14a2e49294b092"
server_key_iv: "961b4d0e12702d3eb5cbc4c8a6f83b49"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.663676 32110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:31.676586 32110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:31.679304 32110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:31.679535 32110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:31.679876 32110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:31.680400 32110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:31.680488 32110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.680555 32110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:31.680604 32110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.687992 32110 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:39281
I20260925 10:05:31.688686 32110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:05:31.690007 32240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:39281 every 8 connection(s)
I20260925 10:05:31.691349 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32110
I20260925 10:05:31.691465 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:05:31.691708 19643 external_mini_cluster.cc:1428] Setting key 2107624246bc5dd7263e88ceb8be9ab8
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:31.703188 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.690861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.194:34063)
0925 10:05:31.691095 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:31.691152 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:31.691934 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:31.692337 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:31.692343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:31.692627 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:31.692978 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.692986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.694142 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.694143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.695626 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.695630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.696340 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.696341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.696520 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.697229 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:31.697241 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:31.699153 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:31.701309 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.701319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.701322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.701686 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.701946 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.701947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.701947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.702097 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.702419 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:31.702425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:31.702608 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 10:05:31.702725 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:05:31.702913 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":132,"threads_started":1}
I20260925 10:05:31.703675 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.690891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:34063 (local address 127.19.46.254:45063)
0925 10:05:31.690950 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:31.690953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:31.691263 (+ 310us) server_negotiation.cc:407] Connection header received
0925 10:05:31.692080 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:31.692081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:31.692099 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:31.692131 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:31.693114 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.693999 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.695769 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.696205 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.696316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.699314 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:31.699316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:31.699317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:31.699327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:31.701197 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.701790 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.701791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.701792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.701858 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.702174 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.702175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.702175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.702327 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:31.702390 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:31.703483 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0925 10:05:31.703587 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:05:31.704104 32241 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:31.704249 32241 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:31.704568 32241 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:31.705143 31890 ts_manager.cc:194] Registered new tserver with Master: ee01d250c30f401c9adeb2e8ae65936e (127.19.46.194:39281)
I20260925 10:05:31.705753 31890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:34063
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:05:31.741076 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:31.847008 32247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:31.847280 32247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:31.847322 32247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:31.848904 32247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:31.848997 32247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:31.849052 32247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:05:31.851277 32247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:31.851966 32247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:31.852629 32247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:31.859155 32252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.859543 32253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:31.859156 32255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:31.860246 32247 server_base.cc:1034] running on GCE node
I20260925 10:05:31.860630 32247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:31.860954 32247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:31.862197 32247 hybrid_clock.cc:648] HybridClock initialized: now 1790330731862183 us; error 30 us; skew 500 ppm
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:31.867913 32247 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:05:31.868858 32247 webserver.cc:460] Webserver started at http://127.19.46.195:38339/ using document root <none> and password file <none>
I20260925 10:05:31.869171 32247 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:31.869557 32247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:31.869791 32247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:31.887387 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "cbfb6994b0d3400bb91f5f4fc1183d84"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "ad72c2f435ad695ed0f6fddc2e9cd653"
server_key_iv: "39f50d12d46b67b77be334e9c1a2110f"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.887949 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "cbfb6994b0d3400bb91f5f4fc1183d84"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "ad72c2f435ad695ed0f6fddc2e9cd653"
server_key_iv: "39f50d12d46b67b77be334e9c1a2110f"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.893675 32247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:05:31.913721 32262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.914026 32247 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260925 10:05:31.914147 32247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "cbfb6994b0d3400bb91f5f4fc1183d84"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "ad72c2f435ad695ed0f6fddc2e9cd653"
server_key_iv: "39f50d12d46b67b77be334e9c1a2110f"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:31.914312 32247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:31.945384 32247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:31.948324 32247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:31.948518 32247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:31.948799 32247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:31.949232 32247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:31.949297 32247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.949361 32247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:31.949385 32247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:31.956866 32247 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:34645
I20260925 10:05:31.957423 32247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:05:31.958706 32375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:34645 every 8 connection(s)
I20260925 10:05:31.960489 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32247
I20260925 10:05:31.960629 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:05:31.960843 19643 external_mini_cluster.cc:1428] Setting key 8758e8de1f874374fadcd7f604b6fc79
Sep 25 10:05:31 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330731, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:31.971901 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.958530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:59025 (local address 127.19.46.254:45063)
0925 10:05:31.958584 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:31.958586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:31.959871 (+ 1285us) server_negotiation.cc:407] Connection header received
0925 10:05:31.960741 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:31.960743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:31.960759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:31.960792 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:31.962033 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.962577 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.963667 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.963892 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.963980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.966334 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:31.966336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:31.966336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:31.966346 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:31.969231 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.970375 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.970377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.970377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.970493 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:31.971031 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:31.971032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:31.971033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:31.971216 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:31.971313 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:31.971657 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:05:31.971775 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:05:31.972163 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:31.958540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.195:59025)
0925 10:05:31.959714 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:31.959767 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:31.960601 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:31.961015 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:31.961020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:31.961536 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:31.961908 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.961914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.962680 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.962681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.963570 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:31.963572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:31.964005 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:31.964006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:31.964135 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:31.964739 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:31.964748 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:31.966221 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:31.969420 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.969512 (+ 92us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.969517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.970175 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.970647 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:31.970650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:31.970650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:31.970852 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:31.971317 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:31.971325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:31.971484 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 10:05:31.971583 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:05:31.971832 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":82,"threads_started":1}
I20260925 10:05:31.972787 32376 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:31.972918 32376 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:31.973116 32376 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:31.973613 31890 ts_manager.cc:194] Registered new tserver with Master: cbfb6994b0d3400bb91f5f4fc1183d84 (127.19.46.195:34645)
I20260925 10:05:31.974148 31890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:59025
I20260925 10:05:31.984994 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:05:31.985721 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 31972
I20260925 10:05:31.991519 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32110
I20260925 10:05:31.997182 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32247
I20260925 10:05:32.003039 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 31858
I20260925 10:05:32.012519 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45063
--webserver_interface=127.19.46.254
--webserver_port=38299
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:45063
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:32.118260 32379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:32.118623 32379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:32.118698 32379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:32.121220 32379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:32.121306 32379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:32.121323 32379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:32.121336 32379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:32.121349 32379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:32.124215 32379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:45063
--ranger_config_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=38299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:32.124577 32379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:32.125058 32379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:32.132028 32387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.132026 32385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.132159 32384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:32.132722 32379 server_base.cc:1034] running on GCE node
I20260925 10:05:32.133119 32379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:32.133599 32379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:32.134857 32379 hybrid_clock.cc:648] HybridClock initialized: now 1790330732134820 us; error 52 us; skew 500 ppm
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:32.138176 32379 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:32.138813 32379 webserver.cc:460] Webserver started at http://127.19.46.254:38299/ using document root <none> and password file <none>
I20260925 10:05:32.139091 32379 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:32.139185 32379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:32.212486 32379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:32.213546 32395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.213788 32379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:32.213932 32379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "a9c2066058804c5d8b9d1ac44451b9eb"
format_stamp: "Formatted at 2026-09-25 10:05:30 on dist-test-slave-f9rl"
server_key: "81613a7a5b148b2b103008dc681ecd9f"
server_key_iv: "5eff033f618eb49011f7593d6f939667"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:32.214560 32379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:32.225718 32379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:32.228963 32379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:32.229228 32379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:32.235347 32379 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:45063
I20260925 10:05:32.235461 32447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:45063 every 8 connection(s)
I20260925 10:05:32.235921 32379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:32.238143 32448 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:05:32.238837 32448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Bootstrap starting.
I20260925 10:05:32.242125 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32379
I20260925 10:05:32.242551 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:33415
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:45063
--webserver_port=34013
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:32.248847 32448 log.cc:824] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:32.250883 32448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:05:32.251260 32448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Bootstrap complete.
I20260925 10:05:32.255385 32448 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:32.255699 32448 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c2066058804c5d8b9d1ac44451b9eb, State: Initialized, Role: FOLLOWER
I20260925 10:05:32.256014 32448 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:32.256114 32448 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:32.256172 32448 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:32.256224 32448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:05:32.258291 32448 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:32.258687 32448 leader_election.cc:302] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c2066058804c5d8b9d1ac44451b9eb; no voters:
I20260925 10:05:32.259204 32448 leader_election.cc:288] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:05:32.259243 32452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:05:32.259366 32452 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [term 2 LEADER]: Becoming Leader. State: Replica: a9c2066058804c5d8b9d1ac44451b9eb, State: Running, Role: LEADER
I20260925 10:05:32.259469 32452 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } }
I20260925 10:05:32.259583 32448 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:32.259739 32452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9c2066058804c5d8b9d1ac44451b9eb. Latest consensus state: current_term: 2 leader_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } } }
I20260925 10:05:32.259753 32453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c2066058804c5d8b9d1ac44451b9eb" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 45063 } } }
I20260925 10:05:32.259869 32452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:32.260015 32453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:32.260233 32457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:32.260942 32457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:05:32.261093 32457 catalog_manager.cc:1259] Loaded cluster ID: 2d352319743f4f8e977e70ca1544f4dc
I20260925 10:05:32.261137 32457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:05:32.261641 32448 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:05:32.262423 32457 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:32.262625 32457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Loaded TSK: 0
I20260925 10:05:32.262842 32457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:05:32.393839 32450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:32.394388 32450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:32.394440 32450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:32.397394 32450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:32.397559 32450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:32.397622 32450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:05:32.401700 32450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:33415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=34013
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:32.402154 32450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:32.402601 32450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:32.412050 32486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.411773 32488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.412513 32485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:32.413479 32450 server_base.cc:1034] running on GCE node
I20260925 10:05:32.413774 32450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:32.414108 32450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:32.417718 32450 hybrid_clock.cc:648] HybridClock initialized: now 1790330732417697 us; error 162 us; skew 500 ppm
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:32.421229 32450 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:05:32.422010 32450 webserver.cc:460] Webserver started at http://127.19.46.193:34013/ using document root <none> and password file <none>
I20260925 10:05:32.422291 32450 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:32.422391 32450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:32.441278 32450 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:05:32.454119 32497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.454587 32450 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260925 10:05:32.454711 32450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "975b82f728c94884bdf0d71272e9d40e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "c447ad10b4f45bfc71a65d352d06d2e3"
server_key_iv: "19e380793a9c9d60bf51ab0281461bfc"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:32.455271 32450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:32.481735 32450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:32.484200 32450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:32.484421 32450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:32.484692 32450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:32.485066 32450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:32.485116 32450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.485155 32450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:32.485179 32450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.495648 32450 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:33415
I20260925 10:05:32.496119 32450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:05:32.497594 32611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:33415 every 8 connection(s)
I20260925 10:05:32.503545 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32450
I20260925 10:05:32.503897 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:39281
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:45063
--webserver_port=36425
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:32.511389 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.497251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.193:33603)
0925 10:05:32.497492 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:32.497544 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:32.498396 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:32.498969 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:32.498974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:32.499207 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:32.499434 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.499438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.501258 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.501259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.502765 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.502770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.503796 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.503798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.503958 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.504556 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:32.504577 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:32.506343 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:32.508938 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.508948 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.508951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.509316 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.509697 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.509698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.509699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.509872 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.510526 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:32.510532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:32.510720 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 10:05:32.510842 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:05:32.511048 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260925 10:05:32.512042 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.497335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:33603 (local address 127.19.46.254:45063)
0925 10:05:32.497916 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:32.497924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:32.497994 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:05:32.498681 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:32.498686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:32.498734 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:32.498928 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:32.499552 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.501099 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.502932 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.503543 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.503683 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.506509 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:32.506514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:32.506514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:32.506527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:32.508790 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.509468 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.509472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.509473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.509606 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.509988 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.509989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.509989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.510427 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:32.510536 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:32.511632 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0925 10:05:32.511790 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":119,"threads_started":1}
I20260925 10:05:32.512878 32612 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:32.512985 32612 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.513244 32612 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.514299 32612 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:45063 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:05:32.514391 32612 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.514483 32612 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:32.515166 32612 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.515272 32612 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.515609 32612 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:05:32.620456 32616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:32.620767 32616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:32.620805 32616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:32.622797 32616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:32.622890 32616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:32.622939 32616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:05:32.625192 32616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:39281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=36425
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:32.625594 32616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:32.625902 32616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:32.631372 32622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.631373 32621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.631372 32624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:32.632683 32616 server_base.cc:1034] running on GCE node
I20260925 10:05:32.632933 32616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:32.633235 32616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:32.634459 32616 hybrid_clock.cc:648] HybridClock initialized: now 1790330732634447 us; error 30 us; skew 500 ppm
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:32.638823 32616 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:05:32.639405 32616 webserver.cc:460] Webserver started at http://127.19.46.194:36425/ using document root <none> and password file <none>
I20260925 10:05:32.639667 32616 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:32.639752 32616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:32.656319 32616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:32.657418 32632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.657665 32616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:32.657773 32616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "ee01d250c30f401c9adeb2e8ae65936e"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "0b2d48686c9677fd0c14a2e49294b092"
server_key_iv: "961b4d0e12702d3eb5cbc4c8a6f83b49"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:32.658217 32616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:32.683403 32616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:32.686077 32616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:32.686276 32616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:32.686573 32616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:32.686909 32616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:32.686967 32616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.687011 32616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:32.687037 32616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.695292 32616 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:39281
I20260925 10:05:32.695485 32746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:39281 every 8 connection(s)
I20260925 10:05:32.695850 32616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:05:32.701795 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32616
I20260925 10:05:32.702204 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.46.212:39889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:34645
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:45063
--webserver_port=38339
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:32.708665 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.696727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52321 (local address 127.19.46.254:45063)
0925 10:05:32.696793 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:32.696795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:32.697056 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:05:32.697970 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:32.697972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:32.697988 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:32.698061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:32.698935 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.699691 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.701424 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.701901 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.701983 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.704717 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:32.704718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:32.704719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:32.704727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:32.706956 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.707513 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.707515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.707515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.707577 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.707858 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.707859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.707859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.708007 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:32.708054 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:32.708465 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 10:05:32.708564 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:05:32.709430 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.696694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.194:52321)
0925 10:05:32.696880 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:32.696934 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:32.697819 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:32.698161 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:32.698166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:32.698472 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:32.698798 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.698806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.699813 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.699814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.701227 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.701231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.702089 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.702090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.702246 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.703008 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:32.703027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:32.704590 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:32.707063 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.707072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.707075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.707414 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.707650 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.707650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.707651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.707785 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.708104 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:32.708111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:32.708283 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 10:05:32.708989 (+ 706us) client_negotiation.cc:240] Negotiation successful
0925 10:05:32.709169 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 10:05:32.710152 32747 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:32.710388 32747 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.710644 32747 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.711395 32747 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:45063 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:05:32.711520 32747 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.711627 32747 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:32.712031 32747 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.712143 32747 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.712548 32747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:05:32.846071 32750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:32.846458 32750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:32.846520 32750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:32.848618 32750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:32.848703 32750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:32.848755 32750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:05:32.851951 32750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.46.212:39889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:34645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=38339
--tserver_master_addrs=127.19.46.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:32.852316 32750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:32.852658 32750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:32.859351 32756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:32.859915 32755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:32.860317 32750 server_base.cc:1034] running on GCE node
W20260925 10:05:32.860646 32758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:32.861028 32750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:32.861415 32750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:32.862658 32750 hybrid_clock.cc:648] HybridClock initialized: now 1790330732862624 us; error 52 us; skew 500 ppm
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:32.866969 32750 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:05:32.868144 32750 webserver.cc:460] Webserver started at http://127.19.46.195:38339/ using document root <none> and password file <none>
I20260925 10:05:32.868536 32750 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:32.868643 32750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:32.890443 32750 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:05:32.891911 32766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.892293 32750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:32.892491 32750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "cbfb6994b0d3400bb91f5f4fc1183d84"
format_stamp: "Formatted at 2026-09-25 10:05:31 on dist-test-slave-f9rl"
server_key: "ad72c2f435ad695ed0f6fddc2e9cd653"
server_key_iv: "39f50d12d46b67b77be334e9c1a2110f"
server_key_version: "kuduclusterkey@0"
I20260925 10:05:32.893170 32750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:32.947791 32750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:32.950435 32750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:32.950620 32750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:32.950914 32750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:32.951305 32750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:32.951362 32750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.951403 32750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:32.951429 32750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:32.959829 32750 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:34645
I20260925 10:05:32.960376 32750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:05:32.960973 413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:34645 every 8 connection(s)
I20260925 10:05:32.962222 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 32750
Sep 25 10:05:32 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330732, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:32.972985 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.961654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:33685 (local address 127.19.46.254:45063)
0925 10:05:32.961734 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:32.961736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:32.961991 (+ 255us) server_negotiation.cc:407] Connection header received
0925 10:05:32.962949 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:32.962951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:32.962962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:32.963005 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:32.963624 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.964212 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.965456 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.965756 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.965811 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.968425 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:32.968427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:32.968427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:32.968436 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:32.970509 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.971121 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.971122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.971122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.971184 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.972181 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.972183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.972183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.972321 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:32.972382 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:32.972770 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 10:05:32.972879 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:05:32.973698 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.961338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:45063 (local address 127.19.46.195:33685)
0925 10:05:32.961707 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:32.961753 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:32.962832 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:32.963091 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:32.963095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:32.963300 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:32.963521 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.963525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.964324 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.964325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.965308 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:32.965311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.965931 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.965932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:32.966076 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.966729 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:32.966747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:32.968295 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:32.970648 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.970661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.970664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.971024 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.971958 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:32.971959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:32.971959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:32.972089 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:32.972408 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:32.972415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:32.972597 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 10:05:32.973217 (+ 620us) client_negotiation.cc:240] Negotiation successful
0925 10:05:32.973420 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260925 10:05:32.974210 414 heartbeater.cc:347] Connected to a master server at 127.19.46.254:45063
I20260925 10:05:32.974325 414 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.974538 414 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.975342 414 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:45063 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:05:32.975419 414 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.975493 414 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:32.975919 414 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:32.975993 414 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
W20260925 10:05:32.976275 414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:05:32.982488 417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:32.962594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.19.46.254:45063)
0925 10:05:32.962726 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:32.962728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:32.964319 (+ 1591us) server_negotiation.cc:407] Connection header received
0925 10:05:32.964339 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:32.964340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:32.964352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:32.964414 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:32.968100 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.969600 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.972203 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:32.973001 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:32.973276 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:32.978642 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:32.978644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:32.978645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:32.978656 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:32.980871 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.981500 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.981501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.981502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.981572 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:32.981819 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:32.981820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:32.981820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:32.981968 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:32.982200 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:32.982241 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 10:05:32.982346 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260925 10:05:33.515810 32612 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:33.515995 32612 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:33.516443 32412 ts_manager.cc:194] Registered new tserver with Master: 975b82f728c94884bdf0d71272e9d40e (127.19.46.193:33415)
I20260925 10:05:33.517161 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:33603
Sep 25 10:05:33 dist-test-slave-f9rl krb5kdc[31411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330733, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:33.712772 32747 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:33.712962 32747 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:33.713330 32412 ts_manager.cc:194] Registered new tserver with Master: ee01d250c30f401c9adeb2e8ae65936e (127.19.46.194:39281)
I20260925 10:05:33.713891 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:52321
I20260925 10:05:33.976490 414 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:33.976676 414 heartbeater.cc:511] Master 127.19.46.254:45063 requested a full tablet report, sending...
I20260925 10:05:33.977154 32412 ts_manager.cc:194] Registered new tserver with Master: cbfb6994b0d3400bb91f5f4fc1183d84 (127.19.46.195:34645)
I20260925 10:05:33.977738 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:33685
I20260925 10:05:33.982298 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:05:33.996338 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:33.983358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.19.46.254:45063)
0925 10:05:33.983793 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:33.983796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:33.983957 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:05:33.985608 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:33.985609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:33.985628 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:33.985681 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:33.987193 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:33.988010 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:33.989226 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:33.989585 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:33.989658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:33.991376 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:33.991378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:33.991378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:33.991388 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:33.994870 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:33.995393 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:33.995394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:33.995394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:33.995459 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:33.995658 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:33.995659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:33.995659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:33.995813 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:33.995932 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:33.996117 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:05:33.996219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":74,"threads_started":1}
I20260925 10:05:33.998986 32412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:05:34.001546 32412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:05:34.010710 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33415 (local address 127.0.0.1:50550)
0925 10:05:34.006096 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.006141 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:34.006212 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:34.006483 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:34.006486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:34.006489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:34.006586 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.006589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.007903 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.007904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.009241 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.009244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.010374 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.010375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.010462 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.010477 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:05:34.010553 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:05:34.010615 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":6,"thread_start_us":111,"threads_started":1}
I20260925 10:05:34.011215 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.19.46.193:33415)
0925 10:05:34.006021 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.006028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.006297 (+ 269us) server_negotiation.cc:407] Connection header received
0925 10:05:34.006338 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.006341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.006359 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.006459 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:34.006694 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.007794 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.009352 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.010193 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.010270 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.010336 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.011089 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.011140 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":69,"threads_started":1}
I20260925 10:05:34.012192 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:34645 (local address 127.0.0.1:49238)
0925 10:05:34.006577 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.006588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:34.006628 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:34.006992 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:34.006993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:34.006996 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:34.007044 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.007045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.008869 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.008870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.010520 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.010523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.011949 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.011950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.012024 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.012029 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:05:34.012061 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:05:34.012106 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":112,"threads_started":1}
I20260925 10:05:34.012290 32543 tablet_service.cc:1467] Processing CreateTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 (DEFAULT_TABLE table=test-table [id=c913b4fef086457c919a64d0ffe70f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:34.012598 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.19.46.195:34645)
0925 10:05:34.006810 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.006814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.006824 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:05:34.006859 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.006862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.006888 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.007112 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:34.007191 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.008763 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.010627 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.011832 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.012361 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.012435 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.012463 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.012514 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":52,"threads_started":1}
I20260925 10:05:34.012723 32543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b87cd4620f48c6ac57eb94b1bc3791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:34.013329 337 tablet_service.cc:1467] Processing CreateTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 (DEFAULT_TABLE table=test-table [id=c913b4fef086457c919a64d0ffe70f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:34.013689 337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b87cd4620f48c6ac57eb94b1bc3791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:34.017835 447 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: Bootstrap starting.
I20260925 10:05:34.018864 448 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Bootstrap starting.
I20260925 10:05:34.019568 447 tablet_bootstrap.cc:654] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:34.019923 447 log.cc:824] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:34.020424 448 tablet_bootstrap.cc:654] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:34.020717 448 log.cc:824] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:34.022058 447 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: No bootstrap required, opened a new log
I20260925 10:05:34.022140 447 ts_tablet_manager.cc:1397] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:05:34.022821 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.19.46.194:39281)
0925 10:05:34.006737 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.006742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.011578 (+ 4836us) server_negotiation.cc:407] Connection header received
0925 10:05:34.011703 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.011705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.011724 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.017362 (+ 5638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:34.017716 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.019311 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.021123 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.022230 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.022306 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.022404 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.022661 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.022727 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":56,"threads_started":1}
I20260925 10:05:34.023342 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.005475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:39281 (local address 127.0.0.1:54714)
0925 10:05:34.011500 (+ 6025us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.011513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:34.011610 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:34.017574 (+ 5964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:34.017575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:34.017578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:34.017625 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.017627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.019465 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.019467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.020994 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.020998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.022480 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.022481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.022553 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.022557 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:05:34.023189 (+ 632us) client_negotiation.cc:240] Negotiation successful
0925 10:05:34.023256 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6011,"thread_start_us":80,"threads_started":1}
I20260925 10:05:34.024394 32680 tablet_service.cc:1467] Processing CreateTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 (DEFAULT_TABLE table=test-table [id=c913b4fef086457c919a64d0ffe70f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:34.024487 447 raft_consensus.cc:348] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.024657 447 raft_consensus.cc:374] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:34.024698 447 raft_consensus.cc:729] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975b82f728c94884bdf0d71272e9d40e, State: Initialized, Role: FOLLOWER
I20260925 10:05:34.024791 32680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b87cd4620f48c6ac57eb94b1bc3791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:34.024842 447 consensus_queue.cc:260] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.025214 447 ts_tablet_manager.cc:1428] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:05:34.025491 32612 heartbeater.cc:503] Master 127.19.46.254:45063 was elected leader, sending a full tablet report...
I20260925 10:05:34.026921 448 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: No bootstrap required, opened a new log
I20260925 10:05:34.027017 448 ts_tablet_manager.cc:1397] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 10:05:34.029482 448 raft_consensus.cc:348] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.029667 448 raft_consensus.cc:374] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:34.029697 448 raft_consensus.cc:729] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfb6994b0d3400bb91f5f4fc1183d84, State: Initialized, Role: FOLLOWER
I20260925 10:05:34.029850 448 consensus_queue.cc:260] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.030148 448 ts_tablet_manager.cc:1428] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:05:34.030274 414 heartbeater.cc:503] Master 127.19.46.254:45063 was elected leader, sending a full tablet report...
I20260925 10:05:34.033717 452 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: Bootstrap starting.
I20260925 10:05:34.034528 452 tablet_bootstrap.cc:654] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:34.034871 452 log.cc:824] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:34.035943 452 tablet_bootstrap.cc:492] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: No bootstrap required, opened a new log
I20260925 10:05:34.036022 452 ts_tablet_manager.cc:1397] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:05:34.039252 452 raft_consensus.cc:348] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.039469 452 raft_consensus.cc:374] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:34.039503 452 raft_consensus.cc:729] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee01d250c30f401c9adeb2e8ae65936e, State: Initialized, Role: FOLLOWER
I20260925 10:05:34.039664 452 consensus_queue.cc:260] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.040055 32747 heartbeater.cc:503] Master 127.19.46.254:45063 was elected leader, sending a full tablet report...
I20260925 10:05:34.039991 452 ts_tablet_manager.cc:1428] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
Sep 25 10:05:34 dist-test-slave-f9rl krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330733, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for HTTP/127.19.46.212@KRBTEST.COM
I20260925 10:05:34.071123 455 raft_consensus.cc:482] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:34.071266 455 raft_consensus.cc:504] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.071709 455 leader_election.cc:288] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbfb6994b0d3400bb91f5f4fc1183d84 (127.19.46.195:34645), 975b82f728c94884bdf0d71272e9d40e (127.19.46.193:33415)
I20260925 10:05:34.079926 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.072439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:44087 (local address 127.19.46.195:34645)
0925 10:05:34.072545 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.072547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.072890 (+ 343us) server_negotiation.cc:407] Connection header received
0925 10:05:34.073025 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.073026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.073037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.073070 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:34.073311 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.074191 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.078810 (+ 4619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.079441 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.079502 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.079512 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.079770 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.079823 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:05:34.080484 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.071959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:34645 (local address 127.19.46.194:44087)
0925 10:05:34.072800 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.072813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:34.072924 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:34.073148 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:34.073149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:34.073150 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:34.073234 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.073236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.074313 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.074314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.075805 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.075809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.079594 (+ 3785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.079595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.079679 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.079683 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:05:34.080309 (+ 626us) client_negotiation.cc:240] Negotiation successful
0925 10:05:34.080367 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":95,"threads_started":1}
I20260925 10:05:34.082085 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b87cd4620f48c6ac57eb94b1bc3791" candidate_uuid: "ee01d250c30f401c9adeb2e8ae65936e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" is_pre_election: true
I20260925 10:05:34.082283 348 raft_consensus.cc:2454] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee01d250c30f401c9adeb2e8ae65936e in term 0.
I20260925 10:05:34.083062 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.072493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52577 (local address 127.19.46.193:33415)
0925 10:05:34.072581 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.072583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.076033 (+ 3450us) server_negotiation.cc:407] Connection header received
0925 10:05:34.076147 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.076149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.076159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.076191 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:34.076414 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.077217 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.080740 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.081401 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.081506 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.081517 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.082911 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.082983 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:05:34.083316 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.072188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33415 (local address 127.19.46.194:52577)
0925 10:05:34.075966 (+ 3778us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.075979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:34.076066 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:34.076258 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:34.076259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:34.076260 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:34.076335 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.076336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.077335 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.077336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.078676 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:34.078679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.081581 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.081582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:34.081660 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.081664 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:05:34.083150 (+ 1486us) client_negotiation.cc:240] Negotiation successful
0925 10:05:34.083222 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3751,"thread_start_us":74,"threads_started":1}
I20260925 10:05:34.084817 32557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b87cd4620f48c6ac57eb94b1bc3791" candidate_uuid: "ee01d250c30f401c9adeb2e8ae65936e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975b82f728c94884bdf0d71272e9d40e" is_pre_election: true
I20260925 10:05:34.085126 32557 raft_consensus.cc:2454] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee01d250c30f401c9adeb2e8ae65936e in term 0.
I20260925 10:05:34.085331 32633 leader_election.cc:302] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbfb6994b0d3400bb91f5f4fc1183d84, ee01d250c30f401c9adeb2e8ae65936e; no voters:
I20260925 10:05:34.085525 455 raft_consensus.cc:2781] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:05:34.085574 455 raft_consensus.cc:482] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:05:34.085596 455 raft_consensus.cc:3037] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:34.086606 455 raft_consensus.cc:504] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.086789 455 leader_election.cc:288] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [CANDIDATE]: Term 1 election: Requested vote from peers cbfb6994b0d3400bb91f5f4fc1183d84 (127.19.46.195:34645), 975b82f728c94884bdf0d71272e9d40e (127.19.46.193:33415)
I20260925 10:05:34.087443 32557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b87cd4620f48c6ac57eb94b1bc3791" candidate_uuid: "ee01d250c30f401c9adeb2e8ae65936e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975b82f728c94884bdf0d71272e9d40e"
I20260925 10:05:34.087564 32557 raft_consensus.cc:3037] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:34.088608 32557 raft_consensus.cc:2454] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee01d250c30f401c9adeb2e8ae65936e in term 1.
I20260925 10:05:34.088815 32636 leader_election.cc:302] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975b82f728c94884bdf0d71272e9d40e, ee01d250c30f401c9adeb2e8ae65936e; no voters:
I20260925 10:05:34.088878 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b87cd4620f48c6ac57eb94b1bc3791" candidate_uuid: "ee01d250c30f401c9adeb2e8ae65936e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84"
I20260925 10:05:34.088982 348 raft_consensus.cc:3037] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:34.089870 348 raft_consensus.cc:2454] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee01d250c30f401c9adeb2e8ae65936e in term 1.
I20260925 10:05:34.090107 455 raft_consensus.cc:2781] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:34.090291 455 raft_consensus.cc:686] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [term 1 LEADER]: Becoming Leader. State: Replica: ee01d250c30f401c9adeb2e8ae65936e, State: Running, Role: LEADER
I20260925 10:05:34.090420 455 consensus_queue.cc:237] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } }
I20260925 10:05:34.091387 32412 catalog_manager.cc:5495] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e reported cstate change: term changed from 0 to 1, leader changed from <none> to ee01d250c30f401c9adeb2e8ae65936e (127.19.46.194). New cstate: current_term: 1 leader_uuid: "ee01d250c30f401c9adeb2e8ae65936e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee01d250c30f401c9adeb2e8ae65936e" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 39281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 } health_report { overall_health: UNKNOWN } } }
I20260925 10:05:34.114781 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.110928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.19.46.194:39281)
0925 10:05:34.110977 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:34.110979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:34.110987 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:05:34.111037 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:34.111038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:34.111071 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:34.111147 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:05:34.111372 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.112044 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.113284 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:34.113588 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:34.113653 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:34.114089 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:05:34.114309 (+ 220us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:05:34.114393 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:34.114668 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:05:34.114693 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:05:34.117756 32557 raft_consensus.cc:1261] T 08b87cd4620f48c6ac57eb94b1bc3791 P 975b82f728c94884bdf0d71272e9d40e [term 1 FOLLOWER]: Refusing update from remote peer ee01d250c30f401c9adeb2e8ae65936e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:34.117832 348 raft_consensus.cc:1261] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 1 FOLLOWER]: Refusing update from remote peer ee01d250c30f401c9adeb2e8ae65936e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:34.118376 455 consensus_queue.cc:1035] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [LEADER]: Connected to new peer: Peer: permanent_uuid: "975b82f728c94884bdf0d71272e9d40e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:34.118536 458 consensus_queue.cc:1035] T 08b87cd4620f48c6ac57eb94b1bc3791 P ee01d250c30f401c9adeb2e8ae65936e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfb6994b0d3400bb91f5f4fc1183d84" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:34.121908 461 mvcc.cc:205] Tried to move back new op lower bound from 7333194686941040640 to 7333194686834929664. Current Snapshot: MvccSnapshot[applied={T|T < 7333194686941040640}]
I20260925 10:05:34.123947 463 mvcc.cc:205] Tried to move back new op lower bound from 7333194686941040640 to 7333194686834929664. Current Snapshot: MvccSnapshot[applied={T|T < 7333194686941040640 or (T in {7333194686941040640})}]
I20260925 10:05:34.124632 462 mvcc.cc:205] Tried to move back new op lower bound from 7333194686941040640 to 7333194686834929664. Current Snapshot: MvccSnapshot[applied={T|T < 7333194686941040640 or (T in {7333194686941040640})}]
I20260925 10:05:34.126861 32411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:34.127002 32411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:34.127919 32411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9c2066058804c5d8b9d1ac44451b9eb: Sending DeleteTablet for 3 replicas of tablet 08b87cd4620f48c6ac57eb94b1bc3791
I20260925 10:05:34.128463 32543 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:34 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:50550
I20260925 10:05:34.129096 337 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:34 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49238
I20260925 10:05:34.129396 32680 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b87cd4620f48c6ac57eb94b1bc3791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:34 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:54714
I20260925 10:05:34.130235 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32450
W20260925 10:05:34.139446 32399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:33415 (error 108)
I20260925 10:05:34.140244 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32616
I20260925 10:05:34.141635 471 tablet_replica.cc:330] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: stopping tablet replica
I20260925 10:05:34.141928 471 raft_consensus.cc:2229] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:34.142151 471 raft_consensus.cc:2258] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:34.142911 471 ts_tablet_manager.cc:1905] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:34.144999 471 ts_tablet_manager.cc:1918] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:34.145100 471 log.cc:1196] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/08b87cd4620f48c6ac57eb94b1bc3791
I20260925 10:05:34.145414 471 ts_tablet_manager.cc:1939] T 08b87cd4620f48c6ac57eb94b1bc3791 P cbfb6994b0d3400bb91f5f4fc1183d84: Deleting consensus metadata
I20260925 10:05:34.146052 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.145888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:33415 (local address 127.0.0.1:50552)
0925 10:05:34.145954 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.146008 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:05:34.146200 32399 catalog_manager.cc:4584] TS 975b82f728c94884bdf0d71272e9d40e (127.19.46.193:33415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08b87cd4620f48c6ac57eb94b1bc3791: Network error: Client connection negotiation failed: client connection to 127.19.46.193:33415: connect: Connection refused (error 111)
I20260925 10:05:34.146688 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:34.146593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:39281 (local address 127.0.0.1:54734)
0925 10:05:34.146633 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:34.146658 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.194:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 10:05:34.147136 32396 catalog_manager.cc:4841] TS cbfb6994b0d3400bb91f5f4fc1183d84 (127.19.46.195:34645): tablet 08b87cd4620f48c6ac57eb94b1bc3791 (table test-table [id=c913b4fef086457c919a64d0ffe70f53]) successfully deleted
I20260925 10:05:34.147269 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32750
I20260925 10:05:34.156064 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 32379
I20260925 10:05:34.168870 19643 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:05:34.515337 19643 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:05:34.515508 19643 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:05:34.861766 19643 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:05:34.862 UTC [31442] LOG: received smart shutdown request
2026-09-25 10:05:34.869 UTC [31442] LOG: background worker "logical replication launcher" (PID 31451) exited with exit code 1
2026-09-25 10:05:34.870 UTC [31446] LOG: shutting down
2026-09-25 10:05:35.210 UTC [31442] LOG: database system is shut down
2026-09-25T10:05:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (66617 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:05:35 dist-test-slave-f9rl krb5kdc[482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:05:35 dist-test-slave-f9rl krb5kdc[482](info): set up 2 sockets
Sep 25 10:05:35 dist-test-slave-f9rl krb5kdc[482](info): commencing operation
krb5kdc: starting...
W20260925 10:05:36.551445 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:05:36 dist-test-slave-f9rl krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:05:36.706544 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38743
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:36.803344 495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:36.803999 495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:36.804104 495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:36.805763 495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:36.805860 495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:36.805987 495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:36.806041 495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:36.806066 495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:36.809548 495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:36.810061 495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:36.810344 495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:36.816136 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:36.816181 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:36.816165 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:36.816227 495 server_base.cc:1034] running on GCE node
I20260925 10:05:36.816968 495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:36.817364 495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:36.818568 495 hybrid_clock.cc:648] HybridClock initialized: now 1790330736818530 us; error 60 us; skew 500 ppm
Sep 25 10:05:36 dist-test-slave-f9rl krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:36.821764 495 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:36.822321 495 webserver.cc:460] Webserver started at http://127.19.46.254:37999/ using document root <none> and password file <none>
I20260925 10:05:36.822588 495 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:36.822667 495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:36.822898 495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:36.824287 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4e6e651af3454b0c9d1e2b99086e174c"
format_stamp: "Formatted at 2026-09-25 10:05:36 on dist-test-slave-f9rl"
server_key: "a80e1ec3811e65fa2cb037e4d56bc3f0"
server_key_iv: "b8c7cd8c72a6135d71597a8589d23ac3"
server_key_version: "encryptionkey@0"
I20260925 10:05:36.824738 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4e6e651af3454b0c9d1e2b99086e174c"
format_stamp: "Formatted at 2026-09-25 10:05:36 on dist-test-slave-f9rl"
server_key: "a80e1ec3811e65fa2cb037e4d56bc3f0"
server_key_iv: "b8c7cd8c72a6135d71597a8589d23ac3"
server_key_version: "encryptionkey@0"
I20260925 10:05:36.826514 495 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:05:36.827571 510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:36.828006 495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:36.828156 495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4e6e651af3454b0c9d1e2b99086e174c"
format_stamp: "Formatted at 2026-09-25 10:05:36 on dist-test-slave-f9rl"
server_key: "a80e1ec3811e65fa2cb037e4d56bc3f0"
server_key_iv: "b8c7cd8c72a6135d71597a8589d23ac3"
server_key_version: "encryptionkey@0"
I20260925 10:05:36.828284 495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:36.855526 495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:36.858426 495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:36.858640 495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:36.865489 495 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:38743
I20260925 10:05:36.865535 562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:38743 every 8 connection(s)
I20260925 10:05:36.866034 495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:36.867204 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:36.869059 563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:05:36.874728 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 495
I20260925 10:05:36.874910 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:05:36.875456 19643 external_mini_cluster.cc:1428] Setting key 822434e9ab344fd0069a1dceff41e9da
Sep 25 10:05:36 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.253@KRBTEST.COM; defaulting to no policy
I20260925 10:05:36.886818 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.870000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38743 (local address 127.0.0.1:58860)
0925 10:05:36.870251 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:36.870303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:36.870832 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:36.871469 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:36.871473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:36.871900 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:36.872133 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:36.872139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:36.874513 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:36.874516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:36.876993 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:36.877000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:36.878029 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:36.878031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:36.878098 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:36.878894 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:36.878908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:36.881391 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:36.884634 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:36.884644 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:36.884646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:36.884996 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:36.885360 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:36.885361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:36.885362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:36.885558 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:36.886205 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:36.886213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:36.886295 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 10:05:36.886424 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:05:36.886608 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":136,"threads_started":1}
I20260925 10:05:36.886960 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.870135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.19.46.254:38743)
0925 10:05:36.870384 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:36.870388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:36.870423 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:05:36.871171 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:36.871175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:36.871220 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:36.871372 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:36.872342 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:36.874131 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:36.877181 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:36.877907 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:36.878070 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:36.881751 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:36.881772 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:36.881772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:36.881788 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:36.884334 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:36.885122 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:36.885127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:36.885128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:36.885254 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:36.885638 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:36.885639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:36.885640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:36.886047 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:36.886156 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:36.886504 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:05:36.886871 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":252,"server-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260925 10:05:36.888461 563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:05:36.888975 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.888807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32934)
0925 10:05:36.888886 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:36.888947 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 10:05:36.889209 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
I20260925 10:05:36.890584 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.890254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32948)
0925 10:05:36.890350 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:36.890394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.253@KRBTEST.COM" created.
W20260925 10:05:36.891027 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:36649: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
I20260925 10:05:36.919358 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:36649
--webserver_interface=127.19.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:36.940508 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } attempt: 1
I20260925 10:05:36.941465 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.941173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32960)
0925 10:05:36.941291 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:36.941372 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 10:05:36.942166 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:36.942048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32970)
0925 10:05:36.942103 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:36.942141 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:05:36.942351 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:36649: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
I20260925 10:05:37.010654 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } attempt: 2
I20260925 10:05:37.011436 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.011290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32976)
0925 10:05:37.011356 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.011394 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:05:37.012151 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.012047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:32992)
0925 10:05:37.012091 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.012120 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 10:05:37.012449 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:36649: Network error: Client connection negotiation failed: client connection to 127.19.46.253:36649: connect: Connection refused (error 111)
W20260925 10:05:37.017275 570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:37.017635 570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:37.017680 570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:37.019066 570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:37.019160 570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:37.019187 570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:37.019209 570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:37.019236 570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:37.023141 570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:37.023523 570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:37.023788 570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:37.031111 579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:37.032215 576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:37.032217 577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:37.033030 570 server_base.cc:1034] running on GCE node
I20260925 10:05:37.033325 570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:37.034009 570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:37.035234 570 hybrid_clock.cc:648] HybridClock initialized: now 1790330737035167 us; error 85 us; skew 500 ppm
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:37.039997 570 init.cc:377] Logged in from keytab as kudu/127.19.46.253@KRBTEST.COM (short username kudu)
I20260925 10:05:37.040652 570 webserver.cc:460] Webserver started at http://127.19.46.253:39467/ using document root <none> and password file <none>
I20260925 10:05:37.041152 570 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:37.041318 570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:37.041625 570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:37.043452 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data/instance:
uuid: "917ff242d8be46ef82dedc77b04cd511"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "8ff95e76af65e960697481542fd24052"
server_key_iv: "d4a03dfaf19942c26f42f6582d3ab32b"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.043927 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal/instance:
uuid: "917ff242d8be46ef82dedc77b04cd511"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "8ff95e76af65e960697481542fd24052"
server_key_iv: "d4a03dfaf19942c26f42f6582d3ab32b"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.045819 570 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:05:37.046989 586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:37.047474 570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:37.047637 570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "917ff242d8be46ef82dedc77b04cd511"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "8ff95e76af65e960697481542fd24052"
server_key_iv: "d4a03dfaf19942c26f42f6582d3ab32b"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.047858 570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:37.065367 570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:37.068588 570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:37.068831 570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:37.074307 570 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.253:36649
I20260925 10:05:37.074442 638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.253:36649 every 8 connection(s)
I20260925 10:05:37.074838 570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
I20260925 10:05:37.075934 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:37.076797 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 570
I20260925 10:05:37.076946 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-1/wal/instance
I20260925 10:05:37.077173 19643 external_mini_cluster.cc:1428] Setting key a5d3745c854fc34a435eab7e05f86a78
I20260925 10:05:37.078279 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.252@KRBTEST.COM" created.
I20260925 10:05:37.092785 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.079373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.19.46.254:38743)
0925 10:05:37.079471 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.079474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.079835 (+ 361us) server_negotiation.cc:407] Connection header received
0925 10:05:37.080382 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.080384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.080400 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.080480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.081460 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.081975 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.083226 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.083614 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.083714 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.087720 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.087723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.087723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.087737 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.090541 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.091279 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.091281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.091282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.091377 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.091692 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.091693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.091693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.091881 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.091974 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.092456 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.092636 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:05:37.092914 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.079422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38743 (local address 127.0.0.1:58872)
0925 10:05:37.079637 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.079682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.080272 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.080657 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.080661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.081071 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.081311 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.081317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.082092 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.082093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.083064 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.083067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.083765 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.083766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.084148 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.084983 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.084994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.087533 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.090693 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.090710 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.090714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.091166 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.091443 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.091444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.091445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.091611 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.091973 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.091982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.092240 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.092410 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.092652 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":125,"threads_started":1}
I20260925 10:05:37.095695 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:05:37.103724 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } attempt: 3
I20260925 10:05:37.106495 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.096094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:33006)
0925 10:05:37.096184 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.096195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.096243 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.096684 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.096685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.096864 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.096916 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.096918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.098159 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.098160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.098999 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.099002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.099740 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.099742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.099888 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.100450 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.100457 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.102181 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.104640 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.104641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.104642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.105220 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.105507 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.105508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.105508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.105636 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.106079 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.106080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.106123 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.106209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.106354 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:05:37.106592 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.096330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.19.46.253:36649)
0925 10:05:37.096516 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.096522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.096544 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:05:37.096576 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.096577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.096590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.096683 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.096999 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.098020 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.099112 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.099625 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.099707 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.102387 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.102401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.102401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.102415 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.104533 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.105329 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.105333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.105333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.105407 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.105770 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.105771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.105772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.105995 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.106108 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.106257 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.106452 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260925 10:05:37.108656 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:05:37.109151 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.109005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45786)
0925 10:05:37.109062 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.109123 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:05:37.109354 587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
I20260925 10:05:37.109988 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.109872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45792)
0925 10:05:37.109926 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.109958 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:05:37.110121 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:05:37.119380 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.104362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.19.46.253:36649)
0925 10:05:37.104594 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.104597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.104606 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:05:37.104632 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.104633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.104644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.104985 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.105459 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.107226 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.108669 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.109106 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.109229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.113983 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.113988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.113988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.114000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.117390 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.118326 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.118329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.118329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.118419 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.118637 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.118638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.118638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.118798 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.118882 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.119050 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.119152 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260925 10:05:37.119410 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.104269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:33010)
0925 10:05:37.104344 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.104357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.104521 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.104994 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.104995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.105167 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.105302 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.105305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.107681 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.107683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.108544 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.108548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.109388 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.109389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.109553 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.110515 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.110521 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.112982 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.117927 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.117930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.117930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.118175 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.118490 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.118491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.118492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.118559 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.118879 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.118880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.118943 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.119022 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.119148 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 10:05:37.120563 563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:05:37.121163 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.120948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45804)
0925 10:05:37.121020 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.121063 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:05:37.121579 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.121480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45820)
0925 10:05:37.121515 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.121546 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 10:05:37.121703 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
I20260925 10:05:37.123523 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:43047
--webserver_interface=127.19.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:37.152900 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 1
I20260925 10:05:37.153527 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.153277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45830)
0925 10:05:37.153370 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.153480 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:05:37.154028 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.153912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45842)
0925 10:05:37.153973 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.154001 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:05:37.154126 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
I20260925 10:05:37.159387 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 1
I20260925 10:05:37.159961 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.159791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45854)
0925 10:05:37.159883 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.159929 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 10:05:37.160393 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.160285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45858)
0925 10:05:37.160328 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.160372 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:05:37.160512 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
W20260925 10:05:37.213295 647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:37.213562 647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:37.213589 647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:37.215041 647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:37.215098 647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:37.215114 647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:37.215127 647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:37.215139 647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:37.216874 647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38743,127.19.46.253:36649,127.19.46.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:37.217176 647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:37.217499 647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:37.222986 652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:37.223093 647 server_base.cc:1034] running on GCE node
W20260925 10:05:37.223006 653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:37.223007 655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:37.223485 647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:37.223841 647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:37.225039 647 hybrid_clock.cc:648] HybridClock initialized: now 1790330737225007 us; error 38 us; skew 500 ppm
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:37.227939 647 init.cc:377] Logged in from keytab as kudu/127.19.46.252@KRBTEST.COM (short username kudu)
I20260925 10:05:37.228500 647 webserver.cc:460] Webserver started at http://127.19.46.252:34145/ using document root <none> and password file <none>
I20260925 10:05:37.228693 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 2
I20260925 10:05:37.228935 647 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:37.229004 647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:37.229173 647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:37.229213 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.229038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45874)
0925 10:05:37.229127 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.229168 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 10:05:37.229274 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 2
I20260925 10:05:37.229662 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.229547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45888)
0925 10:05:37.229605 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.229634 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:05:37.229909 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.229827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45902)
0925 10:05:37.229861 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.229885 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 10:05:37.230026 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.229942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45904)
0925 10:05:37.229980 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.230001 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 10:05:37.230031 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
W20260925 10:05:37.230114 563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:43047: Network error: Client connection negotiation failed: client connection to 127.19.46.252:43047: connect: Connection refused (error 111)
I20260925 10:05:37.230602 647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data/instance:
uuid: "24607eea1046408abb9794681b563caf"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "9b113d383c7ff56998d9191909c33feb"
server_key_iv: "349c93f233f585f29749bf06ddbbef28"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.230921 647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal/instance:
uuid: "24607eea1046408abb9794681b563caf"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "9b113d383c7ff56998d9191909c33feb"
server_key_iv: "349c93f233f585f29749bf06ddbbef28"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.232532 647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:05:37.233757 662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:37.234097 647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:37.234202 647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "24607eea1046408abb9794681b563caf"
format_stamp: "Formatted at 2026-09-25 10:05:37 on dist-test-slave-f9rl"
server_key: "9b113d383c7ff56998d9191909c33feb"
server_key_iv: "349c93f233f585f29749bf06ddbbef28"
server_key_version: "encryptionkey@0"
I20260925 10:05:37.234295 647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:37.268664 647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:37.271499 647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:37.271688 647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:37.277395 647 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.252:43047
I20260925 10:05:37.277490 715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.252:43047 every 8 connection(s)
I20260925 10:05:37.277920 647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
I20260925 10:05:37.278888 716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:37.280076 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 647
I20260925 10:05:37.280205 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330556283750-19643-0/minicluster-data/master-2/wal/instance
I20260925 10:05:37.280421 19643 external_mini_cluster.cc:1428] Setting key b13b17121655df43b2f3333323e915c1
I20260925 10:05:37.281000 716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:05:37.293165 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.281755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38743 (local address 127.0.0.1:58888)
0925 10:05:37.281919 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.281961 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.282064 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.282246 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.282249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.282553 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.282726 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.282730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.283437 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.283438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.284160 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.284163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.284572 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.284573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.284693 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.285255 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.285266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.286916 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.290561 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.290568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.290571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.290920 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.291390 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.291391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.291392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.291761 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.292121 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.292127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.292488 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.292653 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.292917 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 10:05:37.293808 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.281822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.19.46.254:38743)
0925 10:05:37.281890 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.281892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.282042 (+ 150us) server_negotiation.cc:407] Connection header received
0925 10:05:37.282140 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.282141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.282152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.282211 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.282825 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.283333 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.284258 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.284469 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.284560 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.287648 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.287651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.287651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.287663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.290349 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.291041 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.291042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.291043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.291131 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.291857 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.291859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.291859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.292031 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.292091 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.293406 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.293658 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:05:37.294276 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.281053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.19.46.252:43047)
0925 10:05:37.281246 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.281258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.281296 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:05:37.281998 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.282005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.282061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.282180 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.282827 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.283636 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.284413 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.284733 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.284815 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.287842 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.287854 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.287854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.287862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.291130 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.291984 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.291988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.291988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.292054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.292328 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.292329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.292330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.292619 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.292776 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.294040 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.294150 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 10:05:37.294617 716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:37.303717 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.295002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.19.46.253:36649)
0925 10:05:37.295089 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.295091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.295104 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:05:37.295194 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.295195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.295208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.295267 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.295625 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.296364 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.297205 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.297427 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.297561 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.300767 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.300769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.300769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.300779 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.302436 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.302874 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.302879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.302880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.302934 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.303143 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.303144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.303145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.303273 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.303368 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.303523 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.303612 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:05:37.303717 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.294922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:36649 (local address 127.0.0.1:33020)
0925 10:05:37.295056 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.295067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.295133 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.295292 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.295300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.295482 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.295547 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.295550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.296446 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.296447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.297105 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.297108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.297562 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.297563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.297640 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.298211 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.298217 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.300649 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.302564 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.302565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.302566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.302757 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.303013 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.303014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.303015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.303074 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.303362 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.303363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.303413 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.303496 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.303612 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 10:05:37.304157 716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:05:37.304711 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.295463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.19.46.254:38743)
0925 10:05:37.295501 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.295503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.295510 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:05:37.295579 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.295580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.295605 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.295668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.296128 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.296698 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.297687 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.297945 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.298019 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.300841 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.300843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.300843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.300852 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.303343 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.303781 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.303782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.303783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.303854 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.304083 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.304084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.304084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.304226 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.304466 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.304508 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.304600 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:05:37.314468 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.304464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45924)
0925 10:05:37.304509 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.304520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.304561 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.304748 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.304749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.304928 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.304973 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.304975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.306153 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.306155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.306710 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.306712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.307328 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.307330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.307385 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.308352 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.308359 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.310478 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.312700 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.312702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.312702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.312973 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.313513 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.313514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.313515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.313614 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.314066 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.314068 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.314125 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.314220 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.314345 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:05:37.314790 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.304501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.19.46.252:43047)
0925 10:05:37.304537 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.304539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.304547 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:05:37.304618 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.304619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.304628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.304684 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.305056 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.306030 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.306818 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.307164 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.307329 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.310612 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.310614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.310615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.310623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.312533 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.313093 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.313094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.313095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.313214 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.313788 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.313789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.313790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.313976 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.314042 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.314474 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.314686 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:05:37.315728 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.305882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.19.46.253:36649)
0925 10:05:37.305964 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.305967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.305979 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:05:37.306163 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.306165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.306180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.306267 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.306798 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.307249 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.308031 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.308241 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.308343 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.312211 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.312213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.312213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.312224 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.314302 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.314751 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.314753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.314753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.314813 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.315185 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.315185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.315186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.315308 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.315508 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.315545 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.315626 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:05:37.316628 716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: Bootstrap starting.
I20260925 10:05:37.317901 716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:37.318053 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:05:37.318284 716 log.cc:824] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:37.319272 716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: No bootstrap required, opened a new log
I20260925 10:05:37.321229 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 3
I20260925 10:05:37.321552 716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.321754 716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:37.321817 716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24607eea1046408abb9794681b563caf, State: Initialized, Role: FOLLOWER
I20260925 10:05:37.321972 716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.322458 716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:37.322731 729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.322824 729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:05:37.324810 647 master_runner.cc:186] Error getting master registration for 127.19.46.254:38743: OK, instance_id { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" instance_seqno: 1790330736863899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:05:37.328518 647 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:05:37.329161 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.319868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.19.46.254:38743)
0925 10:05:37.320135 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.320137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.320149 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:05:37.320179 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.320180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.320197 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.320269 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.321217 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.322079 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.323136 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.323431 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.325115 (+ 1684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.325168 (+ 53us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.325169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.325170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.325175 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.327568 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.328036 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.328037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.328037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.328109 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.328335 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.328335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.328336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.328488 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.328906 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.328952 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.329035 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260925 10:05:37.329244 563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } attempt: 3
W20260925 10:05:37.330351 741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:05:37.330420 741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:05:37.332373 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.321866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45940)
0925 10:05:37.321920 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.321932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.321972 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.322941 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.322942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.323160 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.323215 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.323218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.324377 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.324378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.325184 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.325187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.325839 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.325841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.325903 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.326828 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.326834 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.328473 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.331150 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.331151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.331152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.331371 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.331581 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.331582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.331582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.331662 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.332035 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.332036 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.332088 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.332137 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.332244 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:05:37.333194 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.322102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.19.46.252:43047)
0925 10:05:37.322502 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.322504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.322515 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:05:37.322541 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.322542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.322553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.323307 (+ 754us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.323374 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.324265 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.325307 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.325733 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.328257 (+ 2524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.328678 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.328679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.328679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.328687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.331051 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.331450 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.331451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.331452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.331510 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.331761 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.331761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.331762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.331932 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.332913 (+ 981us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.332959 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.333044 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":103,"threads_started":1}
Sep 25 10:05:37 dist-test-slave-f9rl krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330736, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:05:37.335196 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: Bootstrap starting.
I20260925 10:05:37.335937 639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:37.336268 639 log.cc:824] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:37.337344 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.320196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.19.46.252:43047)
0925 10:05:37.320237 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.320240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.328760 (+ 8520us) server_negotiation.cc:407] Connection header received
0925 10:05:37.328879 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.328880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.328893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.328928 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.329846 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.330619 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.331653 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.331935 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.331976 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.333829 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.333831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.333832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.333838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.335746 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.336268 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.336270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.336270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.336340 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.336649 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.336650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.336651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.336800 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.336884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.337134 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.337222 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:05:37.337348 729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:37.337579 729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.337905 729 leader_election.cc:288] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6e651af3454b0c9d1e2b99086e174c (127.19.46.254:38743), 917ff242d8be46ef82dedc77b04cd511 (127.19.46.253:36649)
I20260925 10:05:37.338197 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.320053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.19.46.253:36649)
0925 10:05:37.320333 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.320336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.320345 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:05:37.321296 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.321297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.321309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.321355 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.322280 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.323124 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.324043 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.324329 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.324396 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.325966 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.325968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.325968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.325975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.336666 (+ 10691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.337167 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.337169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.337169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.337227 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.337421 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.337422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.337422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.337609 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.337695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.337968 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.338083 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260925 10:05:37.340250 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.329822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:43047 (local address 127.0.0.1:45956)
0925 10:05:37.329883 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:37.329900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:37.329980 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:37.330182 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:37.330183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:37.330374 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:37.330424 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.330426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.331496 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.331497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.332309 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:37.332312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.332847 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.332848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:37.332907 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.333391 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:37.333397 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:37.336372 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:37.339043 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.339045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.339045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.339284 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.339575 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:37.339575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:37.339576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:37.339641 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:37.339915 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:37.339916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:37.339982 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 10:05:37.340046 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:05:37.340148 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:05:37.340281 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:37.329579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.19.46.252:43047)
0925 10:05:37.329772 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:37.329774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:37.329968 (+ 194us) server_negotiation.cc:407] Connection header received
0925 10:05:37.330055 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:37.330061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:37.330074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:37.330518 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:37.330570 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.331385 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.332421 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:37.332743 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:37.334461 (+ 1718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:37.336524 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:37.336527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:37.336527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:37.336535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:37.338941 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.339365 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.339366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.339367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.339458 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:37.339698 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:37.339699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:37.339699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:37.339854 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:37.339916 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:37.340071 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:05:37.340165 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260925 10:05:37.342351 563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: Bootstrap starting.
I20260925 10:05:37.343194 563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:37.343554 563 log.cc:824] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:37.344542 563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: No bootstrap required, opened a new log
I20260925 10:05:37.346266 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: No bootstrap required, opened a new log
I20260925 10:05:37.347021 563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.347175 563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:37.347213 563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e651af3454b0c9d1e2b99086e174c, State: Initialized, Role: FOLLOWER
I20260925 10:05:37.347370 563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.347766 563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:37.347733 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.347901 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.348968 639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.349141 639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:37.349190 639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917ff242d8be46ef82dedc77b04cd511, State: Initialized, Role: FOLLOWER
I20260925 10:05:37.349340 639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.349756 639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:37.351922 613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24607eea1046408abb9794681b563caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "917ff242d8be46ef82dedc77b04cd511" is_pre_election: true
I20260925 10:05:37.352068 613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24607eea1046408abb9794681b563caf in term 0.
W20260925 10:05:37.352135 769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:05:37.352181 769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:05:37.352341 537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24607eea1046408abb9794681b563caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6e651af3454b0c9d1e2b99086e174c" is_pre_election: true
I20260925 10:05:37.352509 537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24607eea1046408abb9794681b563caf in term 0.
I20260925 10:05:37.352854 663 leader_election.cc:302] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24607eea1046408abb9794681b563caf, 4e6e651af3454b0c9d1e2b99086e174c; no voters:
I20260925 10:05:37.353014 729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:05:37.353134 729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:05:37.353166 729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:37.354370 729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.354540 729 leader_election.cc:288] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [CANDIDATE]: Term 1 election: Requested vote from peers 4e6e651af3454b0c9d1e2b99086e174c (127.19.46.254:38743), 917ff242d8be46ef82dedc77b04cd511 (127.19.46.253:36649)
I20260925 10:05:37.354918 613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24607eea1046408abb9794681b563caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "917ff242d8be46ef82dedc77b04cd511"
I20260925 10:05:37.354918 537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24607eea1046408abb9794681b563caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6e651af3454b0c9d1e2b99086e174c"
I20260925 10:05:37.355020 613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:37.355020 537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:37.356055 537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24607eea1046408abb9794681b563caf in term 1.
I20260925 10:05:37.356055 613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24607eea1046408abb9794681b563caf in term 1.
I20260925 10:05:37.356247 663 leader_election.cc:302] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24607eea1046408abb9794681b563caf, 917ff242d8be46ef82dedc77b04cd511; no voters:
I20260925 10:05:37.356360 729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:37.356539 729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [term 1 LEADER]: Becoming Leader. State: Replica: 24607eea1046408abb9794681b563caf, State: Running, Role: LEADER
I20260925 10:05:37.356647 729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } }
I20260925 10:05:37.357128 729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24607eea1046408abb9794681b563caf. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.357244 729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:37.351115 495 master_runner.cc:186] Error getting master registration for 127.19.46.253:36649: OK, instance_id { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" instance_seqno: 1790330737072504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:05:37.359234 495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:05:37.359530 516 tablet.cc:2369] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:05:37.361594 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.361711 773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:37.361748 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.361968 773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 10:05:37.364464 757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:05:37.364533 757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:05:37.366452 613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [term 1 FOLLOWER]: Refusing update from remote peer 24607eea1046408abb9794681b563caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:37.366477 536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [term 1 FOLLOWER]: Refusing update from remote peer 24607eea1046408abb9794681b563caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:37.366848 729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:37.367120 771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:37.368182 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24607eea1046408abb9794681b563caf. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.368304 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.368350 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24607eea1046408abb9794681b563caf. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.368461 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.370554 729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.370662 729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:37.370666 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.370733 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.370743 774 mvcc.cc:205] Tried to move back new op lower bound from 7333194700250349568 to 7333194700213551104. Current Snapshot: MvccSnapshot[applied={T|T < 7333194700250349568}]
I20260925 10:05:37.370929 777 mvcc.cc:205] Tried to move back new op lower bound from 7333194700250349568 to 7333194700213551104. Current Snapshot: MvccSnapshot[applied={T|T < 7333194700250349568}]
I20260925 10:05:37.374480 773 catalog_manager.cc:1347] Generated new cluster ID: 1ad00631a9244504a66ea7e3a77438a0
I20260925 10:05:37.374542 773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:05:37.378517 771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.378643 771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:37.379074 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24607eea1046408abb9794681b563caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e651af3454b0c9d1e2b99086e174c" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38743 } } peers { permanent_uuid: "917ff242d8be46ef82dedc77b04cd511" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 36649 } } peers { permanent_uuid: "24607eea1046408abb9794681b563caf" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 43047 } } }
I20260925 10:05:37.379177 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:05:37.379896 776 mvcc.cc:205] Tried to move back new op lower bound from 7333194700250349568 to 7333194700213551104. Current Snapshot: MvccSnapshot[applied={T|T < 7333194700250349568}]
I20260925 10:05:37.398290 773 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:05:37.399575 773 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:37.411036 773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24607eea1046408abb9794681b563caf: Generated new TSK 0
I20260925 10:05:37.411367 773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:05:38.352680 769 catalog_manager.cc:1259] Loaded cluster ID: 1ad00631a9244504a66ea7e3a77438a0
I20260925 10:05:38.352769 769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: loading cluster ID for follower catalog manager: success
I20260925 10:05:38.353871 769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: acquiring CA information for follower catalog manager: success
I20260925 10:05:38.354228 769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 917ff242d8be46ef82dedc77b04cd511: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:05:38.365275 757 catalog_manager.cc:1259] Loaded cluster ID: 1ad00631a9244504a66ea7e3a77438a0
I20260925 10:05:38.365360 757 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: loading cluster ID for follower catalog manager: success
I20260925 10:05:38.366511 757 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: acquiring CA information for follower catalog manager: success
I20260925 10:05:38.366925 757 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e6e651af3454b0c9d1e2b99086e174c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:05:38.474057 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 495
I20260925 10:05:38.480477 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 570
I20260925 10:05:38.486238 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 647
2026-09-25T10:05:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3141 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:05:38 dist-test-slave-f9rl krb5kdc[791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:05:38 dist-test-slave-f9rl krb5kdc[791](info): set up 2 sockets
Sep 25 10:05:38 dist-test-slave-f9rl krb5kdc[791](info): commencing operation
krb5kdc: starting...
W20260925 10:05:39.703123 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:05:39 dist-test-slave-f9rl krb5kdc[791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:05:39.841002 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34621
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:34621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:39.938556 804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:39.938864 804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:39.938906 804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:39.941306 804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:39.941572 804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:39.941636 804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:39.941671 804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:39.941704 804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:39.944418 804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:34621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:39.944921 804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:39.945226 804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:05:39.951510 804 server_base.cc:1034] running on GCE node
W20260925 10:05:39.951488 810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:39.951490 812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:39.951490 809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:39.952149 804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:39.952502 804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:39.953689 804 hybrid_clock.cc:648] HybridClock initialized: now 1790330739953674 us; error 49 us; skew 500 ppm
Sep 25 10:05:39 dist-test-slave-f9rl krb5kdc[791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330739, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:39.956512 804 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:39.957124 804 webserver.cc:460] Webserver started at http://127.19.46.254:46365/ using document root <none> and password file <none>
I20260925 10:05:39.957387 804 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:39.957506 804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:39.957679 804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:39.958905 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5"
format_stamp: "Formatted at 2026-09-25 10:05:39 on dist-test-slave-f9rl"
server_key: "74207b093b58c5f227bcbae2dfc36e7c"
server_key_iv: "df7c85b3b990519f9714fcf8c656a037"
server_key_version: "encryptionkey@0"
I20260925 10:05:39.959357 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5"
format_stamp: "Formatted at 2026-09-25 10:05:39 on dist-test-slave-f9rl"
server_key: "74207b093b58c5f227bcbae2dfc36e7c"
server_key_iv: "df7c85b3b990519f9714fcf8c656a037"
server_key_version: "encryptionkey@0"
I20260925 10:05:39.961000 804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:05:39.961998 819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:39.962240 804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:39.962371 804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5"
format_stamp: "Formatted at 2026-09-25 10:05:39 on dist-test-slave-f9rl"
server_key: "74207b093b58c5f227bcbae2dfc36e7c"
server_key_iv: "df7c85b3b990519f9714fcf8c656a037"
server_key_version: "encryptionkey@0"
I20260925 10:05:39.962463 804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:39.990254 804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:39.993067 804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:39.993338 804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:39.998226 804 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:34621
I20260925 10:05:39.998322 871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:34621 every 8 connection(s)
I20260925 10:05:39.998729 804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:39.999656 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:40.002120 872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: Bootstrap starting.
I20260925 10:05:40.002794 872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:40.003113 872 log.cc:824] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:40.003975 872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: No bootstrap required, opened a new log
I20260925 10:05:40.005750 872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } }
I20260925 10:05:40.005897 872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:40.005920 872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d14f7a8e6a45e691a4bd8fb826b6b5, State: Initialized, Role: FOLLOWER
I20260925 10:05:40.006080 872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } }
I20260925 10:05:40.006155 872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:40.006184 872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:40.006214 872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:40.006801 872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } }
I20260925 10:05:40.006888 872 leader_election.cc:302] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89d14f7a8e6a45e691a4bd8fb826b6b5; no voters:
I20260925 10:05:40.007085 872 leader_election.cc:288] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:05:40.007202 875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:40.007398 872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:40.007464 875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [term 1 LEADER]: Becoming Leader. State: Replica: 89d14f7a8e6a45e691a4bd8fb826b6b5, State: Running, Role: LEADER
I20260925 10:05:40.007592 875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } }
I20260925 10:05:40.007790 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 804
I20260925 10:05:40.007891 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:05:40.008113 19643 external_mini_cluster.cc:1428] Setting key 5e0a51231172efd80d9690c8f5e94456
I20260925 10:05:40.008975 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } } }
I20260925 10:05:40.009073 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:40.009310 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d14f7a8e6a45e691a4bd8fb826b6b5. Latest consensus state: current_term: 1 leader_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d14f7a8e6a45e691a4bd8fb826b6b5" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34621 } } }
I20260925 10:05:40.009465 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5 [sys.catalog]: This master's current role is: LEADER
W20260925 10:05:40.010375 892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:05:40.010432 892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:05:40.011811 889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:40.012039 889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:05:40.013918 889 catalog_manager.cc:1347] Generated new cluster ID: 041199e0d7dd4345b3077d16124ec589
I20260925 10:05:40.014035 889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:05:40 dist-test-slave-f9rl krb5kdc[791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:40.020028 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:40.008780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.19.46.254:34621)
0925 10:05:40.008995 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:40.009005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:40.009039 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:05:40.009168 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:40.009188 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:40.009251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:40.010154 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:40.010336 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:40.011546 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:40.012695 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:40.013419 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:40.013590 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:40.016336 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:40.016342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:40.016342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:40.016357 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:40.018266 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:40.018683 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:40.018687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:40.018688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:40.018803 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:40.019028 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:40.019029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:40.019029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:40.019320 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:40.019508 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:40.019581 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:05:40.019848 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260925 10:05:40.021057 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:05:40.023525 889 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:05:40.024101 889 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:40.030444 889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89d14f7a8e6a45e691a4bd8fb826b6b5: Generated new TSK 0
I20260925 10:05:40.030665 889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:05:40.076050 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 804
2026-09-25T10:05:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1588 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:05:40 dist-test-slave-f9rl krb5kdc[900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:05:40 dist-test-slave-f9rl krb5kdc[900](info): set up 2 sockets
Sep 25 10:05:40 dist-test-slave-f9rl krb5kdc[900](info): commencing operation
krb5kdc: starting...
W20260925 10:05:41.244050 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:05:41 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:05:41.390758 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:41.486698 913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:41.486951 913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:41.486980 913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:41.488353 913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:41.488426 913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:41.488443 913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:41.488458 913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:41.488473 913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:41.490309 913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:41.490586 913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:41.491110 913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:41.498522 918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:41.498522 921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:41.498546 919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:41.499182 913 server_base.cc:1034] running on GCE node
I20260925 10:05:41.499540 913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:41.500387 913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:41.501663 913 hybrid_clock.cc:648] HybridClock initialized: now 1790330741501625 us; error 71 us; skew 500 ppm
Sep 25 10:05:41 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330741, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:41.515022 913 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:41.516162 913 webserver.cc:460] Webserver started at http://127.19.46.254:38973/ using document root <none> and password file <none>
I20260925 10:05:41.516762 913 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:41.516934 913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:41.517132 913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:41.518831 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "cdf84cb3484ae4f750a0df13fbe3436d"
server_key_iv: "c0f20a8ea94c3f442231ff568464c55c"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.519309 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "cdf84cb3484ae4f750a0df13fbe3436d"
server_key_iv: "c0f20a8ea94c3f442231ff568464c55c"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.521014 913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:05:41.522575 928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:41.523020 913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:05:41.523208 913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "cdf84cb3484ae4f750a0df13fbe3436d"
server_key_iv: "c0f20a8ea94c3f442231ff568464c55c"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.523317 913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:41.539391 913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:41.544605 913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:41.544951 913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:41.551971 913 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:39743
I20260925 10:05:41.552076 981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:39743 every 8 connection(s)
I20260925 10:05:41.552516 913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:41.553902 982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:41.556668 982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap starting.
I20260925 10:05:41.557367 982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:41.557754 982 log.cc:824] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:41.558238 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 913
I20260925 10:05:41.558594 982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: No bootstrap required, opened a new log
I20260925 10:05:41.558607 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:05:41.559044 19643 external_mini_cluster.cc:1428] Setting key e7d266996260cedd7a8af539d1c96947
I20260925 10:05:41.561508 982 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:41.561726 982 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:41.561769 982 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Initialized, Role: FOLLOWER
I20260925 10:05:41.561931 982 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:41.562021 982 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:41.562075 982 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:41.562127 982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:41.562953 982 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:41.563076 982 leader_election.cc:302] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d1db35185e4cd6a8cb6f19f5de49a9; no voters:
I20260925 10:05:41.563256 982 leader_election.cc:288] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:05:41.563524 982 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:41.563555 987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:41.563921 987 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 LEADER]: Becoming Leader. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Running, Role: LEADER
I20260925 10:05:41.564080 987 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:41.564477 988 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:41.564560 988 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:41.564733 989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d1db35185e4cd6a8cb6f19f5de49a9. Latest consensus state: current_term: 1 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:41.564805 989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
W20260925 10:05:41.565678 1002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:05:41.565773 1002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:05:41.565832 997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:41.566011 997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:05:41.568004 997 catalog_manager.cc:1347] Generated new cluster ID: 9f9a0599cfad4e9db89da5bb11667530
I20260925 10:05:41.568066 997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:05:41 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:41.574469 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:41.559968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.19.46.254:39743)
0925 10:05:41.560215 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:41.560227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:41.560266 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:05:41.561398 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:41.561411 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:41.561517 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:41.561687 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:41.562590 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.563963 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.565253 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.565910 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.566036 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:41.569650 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:41.569659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:41.569660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:41.569684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:41.572368 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:41.572898 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:41.572902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:41.572903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:41.573053 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:41.573304 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:41.573305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:41.573305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:41.573704 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:41.573878 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:41.574033 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:05:41.574192 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":125,"threads_started":1}
I20260925 10:05:41.623991 997 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:05:41.624826 997 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:41.632997 997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Generated new TSK 0
I20260925 10:05:41.633268 997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:05:41.678085 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:41.774755 1008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:41.775010 1008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:41.775038 1008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:41.776336 1008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:41.776400 1008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:41.776432 1008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:05:41.778908 1008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:41.779254 1008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:41.779557 1008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:41.785847 1013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:41.785884 1016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:41.785871 1014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:41.786711 1008 server_base.cc:1034] running on GCE node
I20260925 10:05:41.786986 1008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:41.787369 1008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:41.788573 1008 hybrid_clock.cc:648] HybridClock initialized: now 1790330741788541 us; error 113 us; skew 500 ppm
Sep 25 10:05:41 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330741, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:41.791678 1008 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:05:41.792246 1008 webserver.cc:460] Webserver started at http://127.19.46.193:46535/ using document root <none> and password file <none>
I20260925 10:05:41.792500 1008 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:41.792578 1008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:41.792730 1008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:41.794072 1008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "6f93b213465e4e5eabd42ee094201a47"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "b381cdd77c6cc686cd68d9cc42d23a9e"
server_key_iv: "e1b699a88ac62ce55f4b4716cdaa311e"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.794476 1008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "6f93b213465e4e5eabd42ee094201a47"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "b381cdd77c6cc686cd68d9cc42d23a9e"
server_key_iv: "e1b699a88ac62ce55f4b4716cdaa311e"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.796015 1008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:41.796927 1023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:41.797196 1008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:41.797322 1008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "6f93b213465e4e5eabd42ee094201a47"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "b381cdd77c6cc686cd68d9cc42d23a9e"
server_key_iv: "e1b699a88ac62ce55f4b4716cdaa311e"
server_key_version: "encryptionkey@0"
I20260925 10:05:41.797415 1008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:41.811160 1008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:41.815056 1008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:41.815275 1008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:41.815590 1008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:41.815959 1008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:41.816025 1008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:41.816063 1008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:41.816090 1008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:41.823494 1008 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:39781
I20260925 10:05:41.823720 1136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:39781 every 8 connection(s)
I20260925 10:05:41.824105 1008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:05:41.824838 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1008
I20260925 10:05:41.824997 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:05:41.825292 19643 external_mini_cluster.cc:1428] Setting key 99abe7fd5646ecace742f3e668f810b4
Sep 25 10:05:41 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330741, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
I20260925 10:05:41.844383 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:41.825168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:34591)
0925 10:05:41.825427 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:41.825531 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:41.826730 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:41.827090 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:41.827094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:41.828319 (+ 1225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:41.828702 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:41.828712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.829679 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.829681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:41.831272 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:41.831278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.831871 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.831872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:41.832088 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:41.832934 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:41.832961 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:41.836416 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:41.840413 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:41.840432 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:41.840435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:41.840809 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:41.841290 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:41.841291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:41.841291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:41.842252 (+ 961us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:41.843050 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:41.843094 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:41.843673 (+ 579us) client_negotiation.cc:769] Sending connection context
0925 10:05:41.843874 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 10:05:41.844125 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":126,"threads_started":1}
I20260925 10:05:41.844456 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:41.825217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:34591 (local address 127.19.46.254:39743)
0925 10:05:41.825309 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:41.825312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:41.825713 (+ 401us) server_negotiation.cc:407] Connection header received
0925 10:05:41.826923 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:41.826925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:41.826940 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:41.827017 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:41.828891 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.829541 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.831430 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:41.831670 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:41.831832 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:41.836601 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:41.836603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:41.836603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:41.836613 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:41.840187 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:41.840961 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:41.840964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:41.840964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:41.841055 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:41.842757 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:41.842759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:41.842760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:41.842970 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:41.843057 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:41.844133 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0925 10:05:41.844302 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:05:41.845690 1137 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:41.845890 1137 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:41.846484 1137 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:41.847620 945 ts_manager.cc:194] Registered new tserver with Master: 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781)
I20260925 10:05:41.848994 945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:34591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:05:41.881326 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:41.991652 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:41.992172 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:41.992228 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:41.994148 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:41.994264 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:41.994318 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:05:41.997617 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:41.997974 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:41.998251 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:42.004314 1150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.004822 1148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.004822 1147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.005316 1142 server_base.cc:1034] running on GCE node
I20260925 10:05:42.005666 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:42.006067 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:42.007272 1142 hybrid_clock.cc:648] HybridClock initialized: now 1790330742007167 us; error 122 us; skew 500 ppm
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:42.011163 1142 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:05:42.011881 1142 webserver.cc:460] Webserver started at http://127.19.46.194:33185/ using document root <none> and password file <none>
I20260925 10:05:42.012228 1142 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:42.012432 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:42.012771 1142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:42.015002 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "19ce8fabbc3641f584ef31c501529443"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc4198de0d58bb9fc442f7a13b3f8bd8"
server_key_iv: "72dfdff6fecbf16bbc774543c8cc9066"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.015535 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "19ce8fabbc3641f584ef31c501529443"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc4198de0d58bb9fc442f7a13b3f8bd8"
server_key_iv: "72dfdff6fecbf16bbc774543c8cc9066"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.017239 1142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:05:42.018573 1157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.018852 1142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:05:42.018985 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "19ce8fabbc3641f584ef31c501529443"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc4198de0d58bb9fc442f7a13b3f8bd8"
server_key_iv: "72dfdff6fecbf16bbc774543c8cc9066"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.019093 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:42.037184 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:42.041570 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:42.041884 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:42.042186 1142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:42.042569 1142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:42.042631 1142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.042680 1142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:42.042711 1142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.049680 1142 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:36869
I20260925 10:05:42.049727 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:36869 every 8 connection(s)
I20260925 10:05:42.050091 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:42.059326 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1142
I20260925 10:05:42.059442 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:05:42.059661 19643 external_mini_cluster.cc:1428] Setting key f66bb2f4277291b5ee68dd8b1115a1f2
I20260925 10:05:42.063651 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.051175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:48843 (local address 127.19.46.254:39743)
0925 10:05:42.051318 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.051320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.051570 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:05:42.052116 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.052117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.052128 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.052216 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:42.053394 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.054229 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.055638 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.056005 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.056089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.058919 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:42.058921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:42.058922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:42.058932 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:42.060916 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.062038 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.062040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.062040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.062130 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.062576 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.062578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.062578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.062738 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:42.062823 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.063398 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.063509 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 10:05:42.063966 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.050984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:48843)
0925 10:05:42.051408 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.051450 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.051984 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.052277 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.052281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.052991 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:42.053214 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.053219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.054442 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.054443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.055367 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.055371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.056139 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.056140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.056320 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.056987 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:42.056996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:42.058783 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:42.061092 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.061114 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.061117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.061938 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.062231 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.062233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.062233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.062454 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.062833 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:42.062841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:42.063167 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.063330 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.063563 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260925 10:05:42.064558 1271 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:42.064683 1271 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:42.064927 1271 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:42.065666 945 ts_manager.cc:194] Registered new tserver with Master: 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869)
I20260925 10:05:42.066213 945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:48843
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:05:42.094286 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:42.195611 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:42.195943 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:42.195979 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:42.197743 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:42.197824 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:42.197860 1276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:05:42.200439 1276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:42.201246 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:42.201778 1276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:42.209476 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.209498 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.209583 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.209932 1276 server_base.cc:1034] running on GCE node
I20260925 10:05:42.210183 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:42.210477 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:42.211725 1276 hybrid_clock.cc:648] HybridClock initialized: now 1790330742211711 us; error 33 us; skew 500 ppm
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:42.215397 1276 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:05:42.216249 1276 webserver.cc:460] Webserver started at http://127.19.46.195:34371/ using document root <none> and password file <none>
I20260925 10:05:42.216536 1276 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:42.216614 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:42.216784 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:05:42.218072 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "9c9d166cdc494b75b1f992a148eac40d"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc1e143f23768dd5718effb6383e32a2"
server_key_iv: "009508a4ef07a56e23609b9bc29077af"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.218454 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "9c9d166cdc494b75b1f992a148eac40d"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc1e143f23768dd5718effb6383e32a2"
server_key_iv: "009508a4ef07a56e23609b9bc29077af"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.220046 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:42.221000 1291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.221283 1276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:42.221403 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "9c9d166cdc494b75b1f992a148eac40d"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc1e143f23768dd5718effb6383e32a2"
server_key_iv: "009508a4ef07a56e23609b9bc29077af"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.221580 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:42.241951 1276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:42.245749 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:42.246037 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:42.246325 1276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:42.246739 1276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:42.246814 1276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.246860 1276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:42.246891 1276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.261209 1276 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:36283
I20260925 10:05:42.261579 1404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:36283 every 8 connection(s)
I20260925 10:05:42.262527 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:42.272027 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1276
I20260925 10:05:42.272173 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:05:42.272419 19643 external_mini_cluster.cc:1428] Setting key f6343e15095ca7ff5ba4d59c12141888
I20260925 10:05:42.276551 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.264065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:45147 (local address 127.19.46.254:39743)
0925 10:05:42.264181 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.264184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.264351 (+ 167us) server_negotiation.cc:407] Connection header received
0925 10:05:42.265107 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.265109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.265122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.265193 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:42.266578 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.267121 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.268336 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.268544 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.268636 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.271417 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:42.271418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:42.271419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:42.271427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:42.274333 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.275039 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.275040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.275041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.275121 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.275618 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.275620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.275620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.275829 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:42.275905 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.276383 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.276476 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:05:42.276800 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.263943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:45147)
0925 10:05:42.264140 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.264189 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.264969 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.265363 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.265368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.266165 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:42.266432 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.266438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.267240 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.267241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.268185 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.268188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.268675 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.268676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.268809 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.269618 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:42.269627 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:42.271292 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:42.274498 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.274518 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.274522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.274914 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.275272 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.275273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.275274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.275488 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.276046 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:42.276054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:42.276206 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.276308 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.276526 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 10:05:42.277611 1405 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:42.277799 1405 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:42.278122 1405 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:42.278709 945 ts_manager.cc:194] Registered new tserver with Master: 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
I20260925 10:05:42.279129 945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:45147
I20260925 10:05:42.286470 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:05:42.296778 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.288385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.19.46.254:39743)
0925 10:05:42.288556 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.288558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.288614 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:05:42.288811 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.288812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.288823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.288885 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:42.290069 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.290571 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.291475 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.291663 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.291722 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.293306 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:42.293307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:42.293307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:42.293315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:42.294927 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.295284 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.295285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.295285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.295341 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.295782 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.295785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.295785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.296116 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:42.296321 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.296574 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.296669 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260925 10:05:42.299487 945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:05:42.301764 945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:05:42.310864 1418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.19.46.195:36283)
0925 10:05:42.305904 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.305910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.306091 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:05:42.306165 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.306167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.306190 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.306304 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.306668 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.308042 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.309704 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310264 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310348 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310420 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.310693 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.310749 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":104,"threads_started":1}
I20260925 10:05:42.310885 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36283 (local address 127.0.0.1:48814)
0925 10:05:42.305941 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.305991 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.306077 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.306438 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.306442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.306444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.306553 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.306556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.308205 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.308206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.309590 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.309593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310457 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.310561 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310579 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.310656 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.310806 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":208,"thread_start_us":107,"threads_started":1}
I20260925 10:05:42.310870 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.19.46.194:36869)
0925 10:05:42.306047 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.306073 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.306271 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:05:42.306338 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.306341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.306356 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.306483 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.306687 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.307925 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.309270 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310327 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310412 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310473 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.310736 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.310782 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260925 10:05:42.310993 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.19.46.193:39781)
0925 10:05:42.306167 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.306171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.306305 (+ 134us) server_negotiation.cc:407] Connection header received
0925 10:05:42.306472 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.306475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.306490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.306568 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.306859 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.308080 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.309674 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310400 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310464 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310512 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.310845 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.310898 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":59,"threads_started":1}
I20260925 10:05:42.311017 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:39781 (local address 127.0.0.1:40912)
0925 10:05:42.306230 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.306244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.306341 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.306658 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.306659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.306662 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.306749 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.306752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.308197 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.308198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.309592 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.309593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310543 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.310599 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310606 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.310654 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.310757 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":96,"thread_start_us":201,"threads_started":1}
I20260925 10:05:42.311173 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.305357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36869 (local address 127.0.0.1:58972)
0925 10:05:42.306166 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.306174 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.306244 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.306526 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.306527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.306529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.306577 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.306579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.308073 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.308074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.309154 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.309158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.310513 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.310515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.310599 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.310603 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.310654 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.310765 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":66,"thread_start_us":177,"threads_started":1}
I20260925 10:05:42.312031 1071 tablet_service.cc:1467] Processing CreateTablet for tablet 5f72f06643f340028123f1519bdc8807 (DEFAULT_TABLE table=test-table [id=1e6ed8e5862f40959394578bc18f803b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:42.312456 1071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f72f06643f340028123f1519bdc8807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:42.312387 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 5f72f06643f340028123f1519bdc8807 (DEFAULT_TABLE table=test-table [id=1e6ed8e5862f40959394578bc18f803b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:42.312460 1339 tablet_service.cc:1467] Processing CreateTablet for tablet 5f72f06643f340028123f1519bdc8807 (DEFAULT_TABLE table=test-table [id=1e6ed8e5862f40959394578bc18f803b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:42.312742 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f72f06643f340028123f1519bdc8807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:42.312741 1339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f72f06643f340028123f1519bdc8807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:42.315685 1422 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Bootstrap starting.
I20260925 10:05:42.316480 1424 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Bootstrap starting.
I20260925 10:05:42.316404 1422 tablet_bootstrap.cc:654] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:42.317713 1424 tablet_bootstrap.cc:654] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:42.319360 1422 log.cc:824] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:42.320072 1424 log.cc:824] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:42.320914 1423 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Bootstrap starting.
I20260925 10:05:42.321064 1422 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: No bootstrap required, opened a new log
I20260925 10:05:42.321162 1422 ts_tablet_manager.cc:1397] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260925 10:05:42.321547 1424 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: No bootstrap required, opened a new log
I20260925 10:05:42.321674 1424 ts_tablet_manager.cc:1397] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:05:42.321763 1423 tablet_bootstrap.cc:654] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:42.322366 1423 log.cc:824] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:42.323576 1422 raft_consensus.cc:348] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.323736 1422 raft_consensus.cc:374] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:42.323761 1422 raft_consensus.cc:729] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f93b213465e4e5eabd42ee094201a47, State: Initialized, Role: FOLLOWER
I20260925 10:05:42.323812 1424 raft_consensus.cc:348] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.324000 1424 raft_consensus.cc:374] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:42.324049 1424 raft_consensus.cc:729] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ce8fabbc3641f584ef31c501529443, State: Initialized, Role: FOLLOWER
I20260925 10:05:42.324187 1424 consensus_queue.cc:260] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.324231 1422 consensus_queue.cc:260] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.324674 1424 ts_tablet_manager.cc:1428] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:05:42.324702 1137 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
I20260925 10:05:42.324712 1422 ts_tablet_manager.cc:1428] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 10:05:42.325100 1271 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
W20260925 10:05:42.325242 1138 tablet.cc:2369] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:05:42.325477 1423 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: No bootstrap required, opened a new log
I20260925 10:05:42.325562 1423 ts_tablet_manager.cc:1397] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:05:42.327373 1423 raft_consensus.cc:348] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.327543 1423 raft_consensus.cc:374] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:42.327566 1423 raft_consensus.cc:729] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9d166cdc494b75b1f992a148eac40d, State: Initialized, Role: FOLLOWER
I20260925 10:05:42.327996 1423 consensus_queue.cc:260] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.328563 1405 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
I20260925 10:05:42.328564 1423 ts_tablet_manager.cc:1428] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:05:42.394652 1428 raft_consensus.cc:482] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:42.394799 1428 raft_consensus.cc:504] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.395259 1428 leader_election.cc:288] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283), 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869)
I20260925 10:05:42.399982 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.395615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:59123 (local address 127.19.46.194:36869)
0925 10:05:42.395669 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.395672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.395862 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:05:42.395969 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.395970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.395983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.396040 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.396257 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.397067 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.398924 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.399509 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.399595 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.399604 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.399860 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.399901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:05:42.399982 1418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.395620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38043 (local address 127.19.46.195:36283)
0925 10:05:42.395669 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.395672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.395819 (+ 147us) server_negotiation.cc:407] Connection header received
0925 10:05:42.395896 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.395897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.395907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.396012 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.396332 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.397104 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.398902 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.399509 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.399595 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.399604 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.399860 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.399901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:05:42.400081 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.395520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36283 (local address 127.19.46.193:38043)
0925 10:05:42.395697 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.395710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.395789 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.396059 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.396060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.396064 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.396179 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.396183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.397267 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.397268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.398738 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.398742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.399642 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.399644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.399753 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.399763 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.399843 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.399898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":9,"thread_start_us":94,"threads_started":1}
I20260925 10:05:42.400081 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.395520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36869 (local address 127.19.46.193:59123)
0925 10:05:42.395686 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.395795 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.395853 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.396059 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.396060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.396062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.396179 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.396182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.397207 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.397209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.398833 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.398836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.399644 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.399645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.399753 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.399762 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.399843 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.399898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":36,"thread_start_us":83,"threads_started":1}
I20260925 10:05:42.400547 1359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f72f06643f340028123f1519bdc8807" candidate_uuid: "6f93b213465e4e5eabd42ee094201a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c9d166cdc494b75b1f992a148eac40d" is_pre_election: true
I20260925 10:05:42.400612 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f72f06643f340028123f1519bdc8807" candidate_uuid: "6f93b213465e4e5eabd42ee094201a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ce8fabbc3641f584ef31c501529443" is_pre_election: true
I20260925 10:05:42.400728 1359 raft_consensus.cc:2454] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f93b213465e4e5eabd42ee094201a47 in term 0.
I20260925 10:05:42.400840 1225 raft_consensus.cc:2454] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f93b213465e4e5eabd42ee094201a47 in term 0.
I20260925 10:05:42.401015 1027 leader_election.cc:302] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f93b213465e4e5eabd42ee094201a47, 9c9d166cdc494b75b1f992a148eac40d; no voters:
I20260925 10:05:42.401198 1428 raft_consensus.cc:2781] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:05:42.401260 1428 raft_consensus.cc:482] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:05:42.401299 1428 raft_consensus.cc:3037] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:42.402117 1428 raft_consensus.cc:504] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.402272 1428 leader_election.cc:288] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 1 election: Requested vote from peers 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283), 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869)
I20260925 10:05:42.402451 1359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f72f06643f340028123f1519bdc8807" candidate_uuid: "6f93b213465e4e5eabd42ee094201a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c9d166cdc494b75b1f992a148eac40d"
I20260925 10:05:42.402477 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f72f06643f340028123f1519bdc8807" candidate_uuid: "6f93b213465e4e5eabd42ee094201a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ce8fabbc3641f584ef31c501529443"
I20260925 10:05:42.402549 1225 raft_consensus.cc:3037] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:42.402539 1359 raft_consensus.cc:3037] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:42.403465 1359 raft_consensus.cc:2454] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f93b213465e4e5eabd42ee094201a47 in term 1.
I20260925 10:05:42.403465 1225 raft_consensus.cc:2454] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f93b213465e4e5eabd42ee094201a47 in term 1.
I20260925 10:05:42.403792 1025 leader_election.cc:302] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ce8fabbc3641f584ef31c501529443, 6f93b213465e4e5eabd42ee094201a47; no voters:
I20260925 10:05:42.403916 1428 raft_consensus.cc:2781] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:42.404109 1428 raft_consensus.cc:686] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 LEADER]: Becoming Leader. State: Replica: 6f93b213465e4e5eabd42ee094201a47, State: Running, Role: LEADER
I20260925 10:05:42.404192 1428 consensus_queue.cc:237] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.405220 945 catalog_manager.cc:5495] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "6f93b213465e4e5eabd42ee094201a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } health_report { overall_health: HEALTHY } } }
I20260925 10:05:42.431981 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.428326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.19.46.193:39781)
0925 10:05:42.428384 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.428386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.428496 (+ 110us) server_negotiation.cc:407] Connection header received
0925 10:05:42.428546 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.428547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.428569 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.428643 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:05:42.428969 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.429672 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.430896 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.431115 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.431359 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.431430 (+ 71us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:05:42.431554 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:05:42.431633 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.431781 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.431847 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:05:42.434466 1359 raft_consensus.cc:1261] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Refusing update from remote peer 6f93b213465e4e5eabd42ee094201a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:42.434496 1225 raft_consensus.cc:1261] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Refusing update from remote peer 6f93b213465e4e5eabd42ee094201a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:42.434962 1428 consensus_queue.cc:1035] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:42.435207 1433 consensus_queue.cc:1035] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:42.437958 1436 mvcc.cc:205] Tried to move back new op lower bound from 7333194721007669248 to 7333194720888451072. Current Snapshot: MvccSnapshot[applied={T|T < 7333194721007669248}]
I20260925 10:05:42.438745 1437 mvcc.cc:205] Tried to move back new op lower bound from 7333194721007669248 to 7333194720888451072. Current Snapshot: MvccSnapshot[applied={T|T < 7333194721007669248}]
I20260925 10:05:42.439533 1438 mvcc.cc:205] Tried to move back new op lower bound from 7333194721007669248 to 7333194720888451072. Current Snapshot: MvccSnapshot[applied={T|T < 7333194721007669248 or (T in {7333194721007669248})}]
I20260925 10:05:42.444284 944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:42.444437 944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:42.445958 944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Sending DeleteTablet for 3 replicas of tablet 5f72f06643f340028123f1519bdc8807
I20260925 10:05:42.446410 1339 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f72f06643f340028123f1519bdc8807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:48814
I20260925 10:05:42.446563 1071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f72f06643f340028123f1519bdc8807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:40912
I20260925 10:05:42.446623 1205 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f72f06643f340028123f1519bdc8807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:58972
I20260925 10:05:42.446688 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1008
I20260925 10:05:42.447589 1446 tablet_replica.cc:330] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: stopping tablet replica
I20260925 10:05:42.447588 1448 tablet_replica.cc:330] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: stopping tablet replica
I20260925 10:05:42.447731 1446 raft_consensus.cc:2229] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:42.447778 1448 raft_consensus.cc:2229] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:42.447870 1446 raft_consensus.cc:2258] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:42.447880 1448 raft_consensus.cc:2258] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:42.448364 1448 ts_tablet_manager.cc:1905] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:42.448907 1446 ts_tablet_manager.cc:1905] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:42.450559 1448 ts_tablet_manager.cc:1918] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:42.450687 1448 log.cc:1196] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/5f72f06643f340028123f1519bdc8807
I20260925 10:05:42.451020 1448 ts_tablet_manager.cc:1939] T 5f72f06643f340028123f1519bdc8807 P 19ce8fabbc3641f584ef31c501529443: Deleting consensus metadata
I20260925 10:05:42.451818 1446 ts_tablet_manager.cc:1918] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:42.451931 1446 log.cc:1196] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/5f72f06643f340028123f1519bdc8807
I20260925 10:05:42.452214 1446 ts_tablet_manager.cc:1939] T 5f72f06643f340028123f1519bdc8807 P 9c9d166cdc494b75b1f992a148eac40d: Deleting consensus metadata
I20260925 10:05:42.452262 930 catalog_manager.cc:4841] TS 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): tablet 5f72f06643f340028123f1519bdc8807 (table test-table [id=1e6ed8e5862f40959394578bc18f803b]) successfully deleted
I20260925 10:05:42.452736 932 catalog_manager.cc:4841] TS 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): tablet 5f72f06643f340028123f1519bdc8807 (table test-table [id=1e6ed8e5862f40959394578bc18f803b]) successfully deleted
W20260925 10:05:42.454941 932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:39781 (error 108)
I20260925 10:05:42.455960 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1142
I20260925 10:05:42.456130 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.455891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:39781 (local address 127.0.0.1:40932)
0925 10:05:42.456001 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.456079 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 10:05:42.458904 932 catalog_manager.cc:4584] TS 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f72f06643f340028123f1519bdc8807: Network error: Client connection negotiation failed: client connection to 127.19.46.193:39781: connect: Connection refused (error 111)
I20260925 10:05:42.462472 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1276
I20260925 10:05:42.467625 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 913
I20260925 10:05:42.473982 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--webserver_interface=127.19.46.254
--webserver_port=38973
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:42.571697 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:42.572050 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:42.572085 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:42.574446 1450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:42.574645 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:42.574774 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:42.574808 1450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:42.574973 1450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:42.578099 1450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:42.578471 1450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:42.578791 1450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:42.591584 1456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.591668 1459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.591784 1450 server_base.cc:1034] running on GCE node
W20260925 10:05:42.591662 1457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.592425 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:42.592814 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:42.594036 1450 hybrid_clock.cc:648] HybridClock initialized: now 1790330742594000 us; error 59 us; skew 500 ppm
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:42.597787 1450 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:42.598639 1450 webserver.cc:460] Webserver started at http://127.19.46.254:38973/ using document root <none> and password file <none>
I20260925 10:05:42.599113 1450 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:42.599330 1450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:42.601329 1450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:42.602425 1466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.602782 1450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:42.603044 1450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "cdf84cb3484ae4f750a0df13fbe3436d"
server_key_iv: "c0f20a8ea94c3f442231ff568464c55c"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.603542 1450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:42.624164 1450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:42.627007 1450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:42.627197 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:42.633667 1450 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:39743
I20260925 10:05:42.633778 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:39743 every 8 connection(s)
I20260925 10:05:42.634373 1450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:42.636641 1519 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:05:42.637319 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap starting.
I20260925 10:05:42.641979 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1450
I20260925 10:05:42.642360 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:39781
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=46535
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:42.646778 1519 log.cc:824] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:42.649507 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:05:42.649922 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap complete.
I20260925 10:05:42.652776 1519 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:42.653070 1519 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Initialized, Role: FOLLOWER
I20260925 10:05:42.653270 1519 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:42.653344 1519 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:42.653409 1519 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:42.653488 1519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:05:42.654942 1519 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:42.655050 1519 leader_election.cc:302] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d1db35185e4cd6a8cb6f19f5de49a9; no voters:
I20260925 10:05:42.655289 1519 leader_election.cc:288] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:05:42.655329 1523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:05:42.655449 1523 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 LEADER]: Becoming Leader. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Running, Role: LEADER
I20260925 10:05:42.655541 1523 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:42.655583 1519 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:42.655817 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:42.656059 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:42.655815 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d1db35185e4cd6a8cb6f19f5de49a9. Latest consensus state: current_term: 2 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:42.656318 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:42.656525 1530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:42.657512 1530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:05:42.657639 1530 catalog_manager.cc:1259] Loaded cluster ID: 9f9a0599cfad4e9db89da5bb11667530
I20260925 10:05:42.657894 1530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:05:42.658145 1530 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:05:42.660835 1530 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:05:42.666742 1530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:05:42.667042 1530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Loaded TSK: 0
I20260925 10:05:42.667284 1530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:05:42.749346 1521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:42.749701 1521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:42.749753 1521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:42.751302 1521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:42.751385 1521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:42.751443 1521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:05:42.753346 1521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:39781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=46535
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:42.753691 1521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:42.753963 1521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:42.760672 1547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.760654 1546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.760654 1549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.761269 1521 server_base.cc:1034] running on GCE node
I20260925 10:05:42.761546 1521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:42.761866 1521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:42.763062 1521 hybrid_clock.cc:648] HybridClock initialized: now 1790330742763045 us; error 34 us; skew 500 ppm
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:42.766518 1521 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:05:42.767148 1521 webserver.cc:460] Webserver started at http://127.19.46.193:46535/ using document root <none> and password file <none>
I20260925 10:05:42.767390 1521 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:42.767477 1521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:42.769069 1521 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:42.769866 1556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.770071 1521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:42.770140 1521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "6f93b213465e4e5eabd42ee094201a47"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "b381cdd77c6cc686cd68d9cc42d23a9e"
server_key_iv: "e1b699a88ac62ce55f4b4716cdaa311e"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.770537 1521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:42.783427 1521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:42.786453 1521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:42.786635 1521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:42.786929 1521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:42.787477 1563 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:05:42.789098 1521 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:05:42.789151 1521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:05:42.789209 1521 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:05:42.789943 1521 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:05:42.789987 1521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:42.790032 1563 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Bootstrap starting.
I20260925 10:05:42.798173 1521 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:39781
I20260925 10:05:42.798924 1521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:05:42.799324 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1521
I20260925 10:05:42.799640 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:36869
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=33185
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:42.800886 1670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:39781 every 8 connection(s)
I20260925 10:05:42.803292 1563 log.cc:824] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:42.808808 1563 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:05:42.809197 1563 tablet_bootstrap.cc:492] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Bootstrap complete.
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:42.810571 1563 ts_tablet_manager.cc:1397] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.017s
I20260925 10:05:42.813913 1563 raft_consensus.cc:348] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.814778 1563 raft_consensus.cc:729] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f93b213465e4e5eabd42ee094201a47, State: Initialized, Role: FOLLOWER
I20260925 10:05:42.815047 1563 consensus_queue.cc:260] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:42.815519 1563 ts_tablet_manager.cc:1428] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 10:05:42.817668 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.800177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:38915)
0925 10:05:42.801577 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.801637 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.802373 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.803147 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.803154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.803626 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:42.803829 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.803835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.805047 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.805048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.806196 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.806200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.807320 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.807322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.807744 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.808609 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:42.808620 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:42.810831 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:42.814437 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.814447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.814450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.815283 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.815708 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.815709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.815709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.815848 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.816405 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:42.816413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:42.816587 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.816693 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.817289 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"thread_start_us":120,"threads_started":1}
I20260925 10:05:42.818105 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.800600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38915 (local address 127.19.46.254:39743)
0925 10:05:42.802612 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.802623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.802668 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:05:42.802813 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.802819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.802877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.803023 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:42.803977 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.804900 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.806348 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.807077 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.807198 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.811008 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:42.811015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:42.811015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:42.811031 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:42.814241 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.815412 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.815417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.815418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.815575 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.815935 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.815936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.815936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.816289 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:42.816375 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.817684 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.817869 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1955,"thread_start_us":115,"threads_started":1}
I20260925 10:05:42.819849 1671 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:42.820005 1671 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:42.820259 1671 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:42.821854 1483 ts_manager.cc:194] Registered new tserver with Master: 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781)
I20260925 10:05:42.823396 1483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:38915
I20260925 10:05:42.824563 1671 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
I20260925 10:05:42.831357 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.823328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:39781 (local address 127.0.0.1:40938)
0925 10:05:42.823932 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.823971 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.824058 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.824618 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.824622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.824859 (+ 237us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:42.824946 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.824950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.826458 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.826460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.828250 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.828263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.830875 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.830877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.831038 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.831059 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.831156 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.831250 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":86,"threads_started":1}
I20260925 10:05:42.831845 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.823603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.19.46.193:39781)
0925 10:05:42.823857 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.823863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.824154 (+ 291us) server_negotiation.cc:407] Connection header received
0925 10:05:42.824195 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.824197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.824459 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.824539 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:42.825053 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.826331 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.828589 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.830341 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.830617 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.830795 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.831674 (+ 879us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.831764 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260925 10:05:42.832314 1587 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f72f06643f340028123f1519bdc8807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:42 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:40938
I20260925 10:05:42.833405 1680 tablet_replica.cc:330] stopping tablet replica
I20260925 10:05:42.833599 1680 raft_consensus.cc:2229] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:42.833688 1680 raft_consensus.cc:2258] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:42.835098 1680 ts_tablet_manager.cc:1905] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:42.840459 1680 ts_tablet_manager.cc:1918] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:42.840587 1680 log.cc:1196] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/5f72f06643f340028123f1519bdc8807
I20260925 10:05:42.840865 1680 ts_tablet_manager.cc:1939] T 5f72f06643f340028123f1519bdc8807 P 6f93b213465e4e5eabd42ee094201a47: Deleting consensus metadata
I20260925 10:05:42.841863 1470 catalog_manager.cc:4841] TS 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781): tablet 5f72f06643f340028123f1519bdc8807 (table test-table [id=1e6ed8e5862f40959394578bc18f803b]) successfully deleted
W20260925 10:05:42.914130 1675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:42.914517 1675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:42.914565 1675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:42.916862 1675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:42.916965 1675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:42.917003 1675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:05:42.919963 1675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:36869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=33185
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:42.920419 1675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:42.920749 1675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:42.928324 1686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.928324 1685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:42.928517 1688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:42.929668 1675 server_base.cc:1034] running on GCE node
I20260925 10:05:42.929965 1675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:42.930265 1675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:42.931511 1675 hybrid_clock.cc:648] HybridClock initialized: now 1790330742931485 us; error 32 us; skew 500 ppm
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:42.934988 1675 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:05:42.935491 1675 webserver.cc:460] Webserver started at http://127.19.46.194:33185/ using document root <none> and password file <none>
I20260925 10:05:42.935757 1675 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:42.935824 1675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:42.937542 1675 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:42.938858 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.939155 1675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:42.939268 1675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "19ce8fabbc3641f584ef31c501529443"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc4198de0d58bb9fc442f7a13b3f8bd8"
server_key_iv: "72dfdff6fecbf16bbc774543c8cc9066"
server_key_version: "encryptionkey@0"
I20260925 10:05:42.939749 1675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:42.952965 1675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:42.956837 1675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:42.957069 1675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:42.957394 1675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:42.958145 1675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:42.958207 1675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.958235 1675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:42.958247 1675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:42.965706 1675 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:36869
I20260925 10:05:42.965762 1808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:36869 every 8 connection(s)
I20260925 10:05:42.966169 1675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:05:42.971839 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1675
I20260925 10:05:42.972273 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:36283
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=34371
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:05:42 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330742, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:42.980536 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.967148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:44685 (local address 127.19.46.254:39743)
0925 10:05:42.967228 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:42.967231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:42.967571 (+ 340us) server_negotiation.cc:407] Connection header received
0925 10:05:42.968166 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:42.968167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:42.968190 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:42.968304 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:42.969561 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.970121 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.971426 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.971653 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.971785 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.975982 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:42.975985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:42.975985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:42.975994 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:42.978077 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.979116 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.979118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.979118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.979204 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:42.979669 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:42.979671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:42.979671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:42.979818 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:42.979913 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:42.980358 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 10:05:42.980456 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:05:42.980890 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:42.967074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:44685)
0925 10:05:42.967375 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:42.967416 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:42.968030 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:42.968406 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:42.968410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:42.969092 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:42.969344 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.969349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.970255 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.970256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.971278 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:42.971281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:42.971830 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:42.971831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:42.971999 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:42.972780 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:42.972790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:42.975851 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:42.978247 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.978272 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.978277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.978991 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.979362 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:42.979363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:42.979364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:42.979570 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:42.979908 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:42.979917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:42.980148 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 10:05:42.980313 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 10:05:42.980549 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":73,"threads_started":1}
I20260925 10:05:42.981523 1809 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:42.981635 1809 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:42.981962 1809 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:42.982878 1483 ts_manager.cc:194] Registered new tserver with Master: 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869)
I20260925 10:05:42.983382 1483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:44685
W20260925 10:05:43.071884 1812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:43.072168 1812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:43.072211 1812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:43.073629 1812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:43.073696 1812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:43.073745 1812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:05:43.075510 1812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=34371
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:43.075820 1812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:43.076088 1812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:43.081502 1820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.081394 1817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.081589 1812 server_base.cc:1034] running on GCE node
W20260925 10:05:43.081411 1818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.081948 1812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:43.082257 1812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:43.083423 1812 hybrid_clock.cc:648] HybridClock initialized: now 1790330743083397 us; error 36 us; skew 500 ppm
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:43.086807 1812 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:05:43.087575 1812 webserver.cc:460] Webserver started at http://127.19.46.195:34371/ using document root <none> and password file <none>
I20260925 10:05:43.087913 1812 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:43.088021 1812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:43.096710 1812 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.002s
I20260925 10:05:43.098083 1827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.098340 1812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:43.098454 1812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "9c9d166cdc494b75b1f992a148eac40d"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc1e143f23768dd5718effb6383e32a2"
server_key_iv: "009508a4ef07a56e23609b9bc29077af"
server_key_version: "encryptionkey@0"
I20260925 10:05:43.098964 1812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:43.121973 1812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:43.124521 1812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:43.124655 1812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:43.124869 1812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:43.125207 1812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:43.125243 1812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.125268 1812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:43.125322 1812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.131630 1812 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:36283
I20260925 10:05:43.131723 1941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:36283 every 8 connection(s)
I20260925 10:05:43.132104 1812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:05:43.140777 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1812
I20260925 10:05:43.144960 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.133031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:39897 (local address 127.19.46.254:39743)
0925 10:05:43.133179 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.133181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.133458 (+ 277us) server_negotiation.cc:407] Connection header received
0925 10:05:43.134106 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.134108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.134122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.134198 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:43.135339 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.135897 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.137083 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.137317 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.137410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.140599 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:43.140608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:43.140608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:43.140620 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:43.142776 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:43.143395 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:43.143397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:43.143397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:43.143718 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:43.144156 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:43.144157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:43.144158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:43.144302 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:43.144394 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.144735 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.144853 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 10:05:43.145255 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.132945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:39897)
0925 10:05:43.133271 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.133321 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.133970 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.134302 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:43.134309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:43.134968 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:43.135216 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.135222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.136020 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.136021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.136965 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.136968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.137418 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.137420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.137606 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.138336 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:43.138345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:43.140309 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:43.142903 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:43.142932 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:43.142935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:43.143292 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:43.143847 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:43.143849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:43.143850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:43.144050 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:43.144382 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:43.144390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:43.144560 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:05:43.144702 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:05:43.144914 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":85,"threads_started":1}
I20260925 10:05:43.145946 1942 heartbeater.cc:347] Connected to a master server at 127.19.46.254:39743
I20260925 10:05:43.146085 1942 heartbeater.cc:464] Registering TS with master...
I20260925 10:05:43.146339 1942 heartbeater.cc:511] Master 127.19.46.254:39743 requested a full tablet report, sending...
I20260925 10:05:43.146914 1483 ts_manager.cc:194] Registered new tserver with Master: 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
I20260925 10:05:43.147408 1483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:39897
I20260925 10:05:43.149794 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.141386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.19.46.254:39743)
0925 10:05:43.141585 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.141588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.141728 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:05:43.141816 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.141817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.141839 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.141941 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:43.142732 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.143381 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.144292 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.144523 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.144610 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.146312 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:43.146314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:43.146314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:43.146330 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:43.148351 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:43.148810 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:43.148812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:43.148812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:43.148873 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:43.149128 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:43.149129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:43.149129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:43.149346 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:43.149568 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.149635 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.149712 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260925 10:05:43.151373 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:05:43.155778 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.151936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.19.46.254:39743)
0925 10:05:43.152012 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.152014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.152024 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:05:43.152094 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.152094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.152125 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.152192 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:05:43.152437 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.153264 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.154377 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.154590 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.154662 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.154954 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:05:43.155154 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:05:43.155304 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.155406 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.155450 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:05:43.156307 1483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:05:43.158504 1483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:05:43.161968 1587 tablet_service.cc:1467] Processing CreateTablet for tablet b70f637e55e5496680c896bf8b74baeb (DEFAULT_TABLE table=test-table [id=651318605d7e41f89f776ae253401a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:43.162281 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70f637e55e5496680c896bf8b74baeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:43.164062 1563 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Bootstrap starting.
I20260925 10:05:43.164417 1563 tablet_bootstrap.cc:654] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:43.164994 1563 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: No bootstrap required, opened a new log
I20260925 10:05:43.165030 1563 ts_tablet_manager.cc:1397] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:43.165128 1563 raft_consensus.cc:348] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.165233 1563 raft_consensus.cc:374] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:43.165292 1563 raft_consensus.cc:729] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f93b213465e4e5eabd42ee094201a47, State: Initialized, Role: FOLLOWER
I20260925 10:05:43.165344 1563 consensus_queue.cc:260] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.165513 1563 ts_tablet_manager.cc:1428] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.166785 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.161257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36869 (local address 127.0.0.1:58980)
0925 10:05:43.161326 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.161391 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.161469 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.161885 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:43.161886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:43.161888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:43.161961 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.161963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.163101 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.163102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.165142 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.165148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.166463 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.166464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.166563 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.166570 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:05:43.166627 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:05:43.166695 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":44}
I20260925 10:05:43.166935 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.161272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36283 (local address 127.0.0.1:48816)
0925 10:05:43.161379 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.161390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.161494 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.162010 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:43.162011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:43.162013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:43.162049 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.162050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.163667 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.163669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.165184 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.165187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.166619 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.166621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.166681 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.166684 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:05:43.166716 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:05:43.166766 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260925 10:05:43.167096 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.161579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.19.46.195:36283)
0925 10:05:43.161802 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.161807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.161818 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:05:43.161871 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.161873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.161889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.162006 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:43.162105 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.163409 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.165300 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.166498 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.166829 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.166905 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.166938 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.167005 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":147,"threads_started":1}
I20260925 10:05:43.167177 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.161529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.19.46.194:36869)
0925 10:05:43.161712 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.161718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.161729 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:05:43.161773 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.161775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.161788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.161889 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:43.162132 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.162982 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.165281 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.166316 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.166449 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.166529 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.167020 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.167074 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260925 10:05:43.168041 1743 tablet_service.cc:1467] Processing CreateTablet for tablet b70f637e55e5496680c896bf8b74baeb (DEFAULT_TABLE table=test-table [id=651318605d7e41f89f776ae253401a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:43.168056 1875 tablet_service.cc:1467] Processing CreateTablet for tablet b70f637e55e5496680c896bf8b74baeb (DEFAULT_TABLE table=test-table [id=651318605d7e41f89f776ae253401a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:05:43.168457 1743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70f637e55e5496680c896bf8b74baeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:43.168458 1875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70f637e55e5496680c896bf8b74baeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:05:43.172657 1953 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Bootstrap starting.
I20260925 10:05:43.173667 1952 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Bootstrap starting.
I20260925 10:05:43.174419 1952 tablet_bootstrap.cc:654] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:43.175014 1952 log.cc:824] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:43.175362 1953 tablet_bootstrap.cc:654] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Neither blocks nor log segments found. Creating new log.
I20260925 10:05:43.175958 1953 log.cc:824] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:43.176016 1952 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: No bootstrap required, opened a new log
I20260925 10:05:43.176081 1952 ts_tablet_manager.cc:1397] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:05:43.176810 1953 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: No bootstrap required, opened a new log
I20260925 10:05:43.177134 1953 ts_tablet_manager.cc:1397] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 10:05:43.178980 1952 raft_consensus.cc:348] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.179205 1952 raft_consensus.cc:374] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:43.179256 1952 raft_consensus.cc:729] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9d166cdc494b75b1f992a148eac40d, State: Initialized, Role: FOLLOWER
I20260925 10:05:43.179396 1952 consensus_queue.cc:260] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.179575 1953 raft_consensus.cc:348] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.179769 1953 raft_consensus.cc:374] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:05:43.179785 1942 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
I20260925 10:05:43.179809 1953 raft_consensus.cc:729] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ce8fabbc3641f584ef31c501529443, State: Initialized, Role: FOLLOWER
I20260925 10:05:43.179800 1952 ts_tablet_manager.cc:1428] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 10:05:43.179999 1953 consensus_queue.cc:260] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.180325 1958 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:43.180377 1953 ts_tablet_manager.cc:1428] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 10:05:43.180420 1958 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.180500 1809 heartbeater.cc:503] Master 127.19.46.254:39743 was elected leader, sending a full tablet report...
I20260925 10:05:43.180819 1958 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283), 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781)
I20260925 10:05:43.184804 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.180986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36283 (local address 127.19.46.194:57809)
0925 10:05:43.181039 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.181054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.181117 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.181326 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:43.181327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:43.181329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:43.181409 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.181411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.182587 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.182589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.183532 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.183536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.184328 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.184330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.184462 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.184470 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:05:43.184542 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:05:43.184687 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":23}
I20260925 10:05:43.184726 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.181014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:57809 (local address 127.19.46.195:36283)
0925 10:05:43.181060 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.181063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.181097 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:05:43.181204 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.181205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.181217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.181270 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:43.181552 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.182410 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.183700 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.184156 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.184230 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.184239 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.184584 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.184639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:05:43.184981 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.181184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:39483 (local address 127.19.46.193:39781)
0925 10:05:43.181235 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.181237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.181329 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:05:43.181472 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.181474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.181486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.181522 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:05:43.181738 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.182540 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.183806 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.184398 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.184531 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.184545 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.184823 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.184889 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:05:43.185392 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70f637e55e5496680c896bf8b74baeb" candidate_uuid: "19ce8fabbc3641f584ef31c501529443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c9d166cdc494b75b1f992a148eac40d" is_pre_election: true
I20260925 10:05:43.185627 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.181107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:39781 (local address 127.19.46.194:39483)
0925 10:05:43.181264 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.181276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.181354 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.181597 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:43.181598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:43.181599 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:05:43.181655 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.181657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.182644 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.182645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.183601 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:43.183605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.184593 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.184595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:43.184692 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.184699 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:05:43.184803 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:05:43.184864 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260925 10:05:43.185638 1896 raft_consensus.cc:2454] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ce8fabbc3641f584ef31c501529443 in term 0.
I20260925 10:05:43.186233 1699 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d; no voters:
I20260925 10:05:43.186239 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70f637e55e5496680c896bf8b74baeb" candidate_uuid: "19ce8fabbc3641f584ef31c501529443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f93b213465e4e5eabd42ee094201a47" is_pre_election: true
I20260925 10:05:43.186468 1619 raft_consensus.cc:2454] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ce8fabbc3641f584ef31c501529443 in term 0.
I20260925 10:05:43.186595 1958 raft_consensus.cc:2781] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:05:43.186722 1958 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:05:43.186769 1958 raft_consensus.cc:3037] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:43.187790 1958 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.187991 1958 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [CANDIDATE]: Term 1 election: Requested vote from peers 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283), 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781)
I20260925 10:05:43.188349 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70f637e55e5496680c896bf8b74baeb" candidate_uuid: "19ce8fabbc3641f584ef31c501529443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f93b213465e4e5eabd42ee094201a47"
I20260925 10:05:43.188485 1619 raft_consensus.cc:3037] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:43.188480 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70f637e55e5496680c896bf8b74baeb" candidate_uuid: "19ce8fabbc3641f584ef31c501529443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c9d166cdc494b75b1f992a148eac40d"
I20260925 10:05:43.188621 1896 raft_consensus.cc:3037] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:05:43.190145 1896 raft_consensus.cc:2454] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ce8fabbc3641f584ef31c501529443 in term 1.
I20260925 10:05:43.190250 1619 raft_consensus.cc:2454] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ce8fabbc3641f584ef31c501529443 in term 1.
I20260925 10:05:43.190480 1699 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d; no voters:
I20260925 10:05:43.190630 1958 raft_consensus.cc:2781] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:05:43.190711 1958 raft_consensus.cc:686] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 1 LEADER]: Becoming Leader. State: Replica: 19ce8fabbc3641f584ef31c501529443, State: Running, Role: LEADER
I20260925 10:05:43.190815 1958 consensus_queue.cc:237] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.192024 1482 catalog_manager.cc:5495] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ce8fabbc3641f584ef31c501529443 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "19ce8fabbc3641f584ef31c501529443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } health_report { overall_health: UNKNOWN } } }
I20260925 10:05:43.199916 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.196451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.19.46.194:36869)
0925 10:05:43.196545 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:43.196547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:43.196558 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:05:43.196600 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:43.196601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:43.196621 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:43.196690 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:05:43.196909 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.197648 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.198750 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:43.198979 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:43.199061 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:43.199279 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:05:43.199427 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:05:43.199510 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:43.199611 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:05:43.199633 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:05:43.202361 1619 raft_consensus.cc:1261] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Refusing update from remote peer 19ce8fabbc3641f584ef31c501529443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:43.202479 1896 raft_consensus.cc:1261] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Refusing update from remote peer 19ce8fabbc3641f584ef31c501529443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:05:43.202805 1958 consensus_queue.cc:1035] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:43.202997 1957 consensus_queue.cc:1035] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:05:43.206889 1963 mvcc.cc:205] Tried to move back new op lower bound from 7333194724152868864 to 7333194724110643200. Current Snapshot: MvccSnapshot[applied={T|T < 7333194724152868864}]
I20260925 10:05:43.207474 1964 mvcc.cc:205] Tried to move back new op lower bound from 7333194724152868864 to 7333194724110643200. Current Snapshot: MvccSnapshot[applied={T|T < 7333194724152868864}]
I20260925 10:05:43.209311 1482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:43.209470 1482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:05:43.210981 1482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Sending DeleteTablet for 3 replicas of tablet b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:43.211524 1875 tablet_service.cc:1514] Processing DeleteTablet for tablet b70f637e55e5496680c896bf8b74baeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:43 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:48816
I20260925 10:05:43.211544 1587 tablet_service.cc:1514] Processing DeleteTablet for tablet b70f637e55e5496680c896bf8b74baeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:43 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:40938
I20260925 10:05:43.211686 1680 tablet_replica.cc:330] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: stopping tablet replica
I20260925 10:05:43.211750 1680 raft_consensus.cc:2229] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:43.211815 1680 raft_consensus.cc:2258] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:43.211915 1743 tablet_service.cc:1514] Processing DeleteTablet for tablet b70f637e55e5496680c896bf8b74baeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:05:43 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:58980
I20260925 10:05:43.212185 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1521
I20260925 10:05:43.212230 1973 tablet_replica.cc:330] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: stopping tablet replica
I20260925 10:05:43.212337 1973 raft_consensus.cc:2229] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:05:43.212795 1973 raft_consensus.cc:2258] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:43.212226 1972 tablet_replica.cc:330] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: stopping tablet replica
I20260925 10:05:43.213375 1972 raft_consensus.cc:2229] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:05:43.213380 1973 ts_tablet_manager.cc:1905] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:43.213559 1972 raft_consensus.cc:2258] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:05:43.214063 1972 ts_tablet_manager.cc:1905] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:05:43.215592 1973 ts_tablet_manager.cc:1918] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:43.215672 1973 log.cc:1196] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:43.216001 1973 ts_tablet_manager.cc:1939] T b70f637e55e5496680c896bf8b74baeb P 19ce8fabbc3641f584ef31c501529443: Deleting consensus metadata
I20260925 10:05:43.216238 1972 ts_tablet_manager.cc:1918] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:05:43.216511 1468 catalog_manager.cc:4841] TS 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): tablet b70f637e55e5496680c896bf8b74baeb (table test-table [id=651318605d7e41f89f776ae253401a3c]) successfully deleted
I20260925 10:05:43.216312 1972 log.cc:1196] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:43.216984 1972 ts_tablet_manager.cc:1939] T b70f637e55e5496680c896bf8b74baeb P 9c9d166cdc494b75b1f992a148eac40d: Deleting consensus metadata
I20260925 10:05:43.217459 1470 catalog_manager.cc:4841] TS 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): tablet b70f637e55e5496680c896bf8b74baeb (table test-table [id=651318605d7e41f89f776ae253401a3c]) successfully deleted
W20260925 10:05:43.220167 1470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:39781 (error 108)
I20260925 10:05:43.220458 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1675
I20260925 10:05:43.221122 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.220587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:39781 (local address 127.0.0.1:40952)
0925 10:05:43.221033 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.221077 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
W20260925 10:05:43.222201 1470 catalog_manager.cc:4584] TS 6f93b213465e4e5eabd42ee094201a47 (127.19.46.193:39781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b70f637e55e5496680c896bf8b74baeb: Network error: Client connection negotiation failed: client connection to 127.19.46.193:39781: connect: Connection refused (error 111)
I20260925 10:05:43.227151 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1812
I20260925 10:05:43.232735 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 1450
I20260925 10:05:43.239193 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--webserver_interface=127.19.46.254
--webserver_port=38973
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:43.333081 1975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:43.333379 1975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:43.333421 1975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:43.334705 1975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:05:43.334805 1975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:43.334829 1975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:43.334851 1975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:05:43.334882 1975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:05:43.336529 1975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:43.336892 1975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:43.337164 1975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:43.343886 1983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.343943 1980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.344045 1981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.344388 1975 server_base.cc:1034] running on GCE node
I20260925 10:05:43.344650 1975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:43.345031 1975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:43.346179 1975 hybrid_clock.cc:648] HybridClock initialized: now 1790330743346158 us; error 32 us; skew 500 ppm
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:43.349532 1975 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:05:43.350180 1975 webserver.cc:460] Webserver started at http://127.19.46.254:38973/ using document root <none> and password file <none>
I20260925 10:05:43.350493 1975 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:43.350574 1975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:43.352250 1975 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:43.353428 1990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.353789 1975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:43.353917 1975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "cdf84cb3484ae4f750a0df13fbe3436d"
server_key_iv: "c0f20a8ea94c3f442231ff568464c55c"
server_key_version: "encryptionkey@0"
I20260925 10:05:43.354461 1975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:43.363699 1975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:43.366497 1975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:43.366711 1975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:43.371379 1975 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:39743
I20260925 10:05:43.371450 2042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:39743 every 8 connection(s)
I20260925 10:05:43.371881 1975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:05:43.374003 2043 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:05:43.374907 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap starting.
I20260925 10:05:43.375379 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 1975
I20260925 10:05:43.375676 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:39781
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=46535
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:43.384514 2043 log.cc:824] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:43.387092 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:05:43.387473 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9: Bootstrap complete.
I20260925 10:05:43.389940 2043 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:43.390259 2043 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Initialized, Role: FOLLOWER
I20260925 10:05:43.390481 2043 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:43.390576 2043 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:05:43.390631 2043 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:05:43.390681 2043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:05:43.391765 2043 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:43.391896 2043 leader_election.cc:302] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d1db35185e4cd6a8cb6f19f5de49a9; no voters:
I20260925 10:05:43.392127 2043 leader_election.cc:288] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:05:43.392166 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:05:43.392273 2047 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [term 3 LEADER]: Becoming Leader. State: Replica: f5d1db35185e4cd6a8cb6f19f5de49a9, State: Running, Role: LEADER
I20260925 10:05:43.392416 2043 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:05:43.392444 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } }
I20260925 10:05:43.392779 2047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d1db35185e4cd6a8cb6f19f5de49a9. Latest consensus state: current_term: 3 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:43.392783 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1db35185e4cd6a8cb6f19f5de49a9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39743 } } }
I20260925 10:05:43.392904 2047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:43.392992 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1db35185e4cd6a8cb6f19f5de49a9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:05:43.393258 2056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:05:43.394181 2056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:05:43.394402 2056 catalog_manager.cc:1259] Loaded cluster ID: 9f9a0599cfad4e9db89da5bb11667530
I20260925 10:05:43.394593 2056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:05:43.394889 2056 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f71687879ad google::LogMessage::Fail() at ??:0
@ 0x7f716878a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f71687874de google::LogMessage::Flush() at ??:0
@ 0x7f7168788a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f71706a80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f71706b18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f716a6550d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f716a64a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f716b4ae6db start_thread at ??:0
@ 0x7f716728561f clone at ??:0
W20260925 10:05:43.480216 2045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:43.480517 2045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:43.480566 2045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:43.482093 2045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:43.482170 2045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:43.482221 2045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:05:43.483978 2045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:39781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=46535
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:43.484305 2045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:43.484576 2045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:05:43.490775 2079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.490911 2080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.490891 2045 server_base.cc:1034] running on GCE node
W20260925 10:05:43.491081 2082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.491325 2045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:43.491637 2045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:43.492801 2045 hybrid_clock.cc:648] HybridClock initialized: now 1790330743492782 us; error 51 us; skew 500 ppm
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:43.497546 2045 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:05:43.498170 2045 webserver.cc:460] Webserver started at http://127.19.46.193:46535/ using document root <none> and password file <none>
I20260925 10:05:43.498483 2045 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:43.498560 2045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:43.508497 2045 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.001s
I20260925 10:05:43.509532 2089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.509733 2045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:43.509862 2045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "6f93b213465e4e5eabd42ee094201a47"
format_stamp: "Formatted at 2026-09-25 10:05:41 on dist-test-slave-f9rl"
server_key: "b381cdd77c6cc686cd68d9cc42d23a9e"
server_key_iv: "e1b699a88ac62ce55f4b4716cdaa311e"
server_key_version: "encryptionkey@0"
I20260925 10:05:43.510270 2045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:43.535789 2045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:43.538383 2045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:43.538515 2045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:43.538794 2045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:43.540177 2096 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:05:43.542894 2045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:05:43.543015 2045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.001s sys 0.000s
I20260925 10:05:43.543068 2045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:05:43.544160 2045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:05:43.544262 2045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:05:43.544275 2096 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Bootstrap starting.
I20260925 10:05:43.554060 2045 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:39781
I20260925 10:05:43.554157 2203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:39781 every 8 connection(s)
I20260925 10:05:43.554725 2045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:05:43.559020 2096 log.cc:824] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Log is configured to *not* fsync() on all Append() calls
I20260925 10:05:43.562664 2096 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:05:43.563073 2096 tablet_bootstrap.cc:492] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Bootstrap complete.
I20260925 10:05:43.563336 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2045
I20260925 10:05:43.563577 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:36869
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=33185
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:05:43.564527 2096 ts_tablet_manager.cc:1397] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20260925 10:05:43.566417 2096 raft_consensus.cc:348] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.566843 2096 raft_consensus.cc:729] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f93b213465e4e5eabd42ee094201a47, State: Initialized, Role: FOLLOWER
I20260925 10:05:43.567024 2096 consensus_queue.cc:260] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:43.567396 2096 ts_tablet_manager.cc:1428] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47: Time spent starting tablet: real 0.003s user 0.004s sys 0.002s
W20260925 10:05:43.638913 2206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:05:43.555584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:43447)
0925 10:05:43.555916 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.555959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:43.556476 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:43.638572 (+ 82096us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":122,"threads_started":1}
W20260925 10:05:43.639180 2204 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:39743 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:39743: Client connection negotiation failed: client connection to 127.19.46.254:39743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:05:43.639597 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.639465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:50359)
0925 10:05:43.639527 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.639576 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:05:43.640187 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.640070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:58219)
0925 10:05:43.640123 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.640160 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:05:43.640350 2204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:05:43.663869 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:43.664112 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:43.664140 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:43.665405 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:43.665524 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:43.665586 2208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:05:43.668756 2208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:36869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=33185
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:43.669130 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:43.669417 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:05:43.676072 2208 server_base.cc:1034] running on GCE node
W20260925 10:05:43.676030 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.676029 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.676041 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.676599 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:43.676905 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:43.678110 2208 hybrid_clock.cc:648] HybridClock initialized: now 1790330743678090 us; error 40 us; skew 500 ppm
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:43.681218 2208 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:05:43.681843 2208 webserver.cc:460] Webserver started at http://127.19.46.194:33185/ using document root <none> and password file <none>
I20260925 10:05:43.682210 2208 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:43.682291 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:43.683755 2208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:05:43.684557 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.684942 2208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:05:43.685086 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "19ce8fabbc3641f584ef31c501529443"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc4198de0d58bb9fc442f7a13b3f8bd8"
server_key_iv: "72dfdff6fecbf16bbc774543c8cc9066"
server_key_version: "encryptionkey@0"
I20260925 10:05:43.685578 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:43.713924 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:43.716620 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:43.716825 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:43.717085 2208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:43.717551 2208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:43.717615 2208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.717658 2208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:43.717689 2208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.724222 2208 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:36869
I20260925 10:05:43.724323 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:36869 every 8 connection(s)
I20260925 10:05:43.724684 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:05:43.725998 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.725519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:60421)
0925 10:05:43.725692 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.725788 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260925 10:05:43.726495 2338 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:39743 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:39743: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
I20260925 10:05:43.726862 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.726710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:46889)
0925 10:05:43.726794 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.726834 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 10:05:43.727339 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.727154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:37445)
0925 10:05:43.727281 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.727315 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260925 10:05:43.727530 2338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:05:43.731835 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2208
I20260925 10:05:43.732149 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:36283
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:39743
--webserver_port=34371
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:05:43.823065 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:05:43.823414 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:05:43.823467 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:05:43.824967 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:05:43.825067 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:05:43.825120 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:05:43.827033 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=34371
--tserver_master_addrs=127.19.46.254:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:05:43.827402 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:05:43.827708 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:05:43.835117 2341 server_base.cc:1034] running on GCE node
W20260925 10:05:43.835072 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.835072 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:05:43.835108 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:05:43.835666 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:05:43.836071 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:05:43.837255 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790330743837245 us; error 37 us; skew 500 ppm
Sep 25 10:05:43 dist-test-slave-f9rl krb5kdc[900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:05:43.841183 2341 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:05:43.841883 2341 webserver.cc:460] Webserver started at http://127.19.46.195:34371/ using document root <none> and password file <none>
I20260925 10:05:43.842154 2341 fs_manager.cc:359] Metadata directory not provided
I20260925 10:05:43.842204 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:05:43.843868 2341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:05:43.844960 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.845181 2341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:05:43.845274 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "9c9d166cdc494b75b1f992a148eac40d"
format_stamp: "Formatted at 2026-09-25 10:05:42 on dist-test-slave-f9rl"
server_key: "dc1e143f23768dd5718effb6383e32a2"
server_key_iv: "009508a4ef07a56e23609b9bc29077af"
server_key_version: "encryptionkey@0"
I20260925 10:05:43.845954 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:05:43.869939 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:05:43.872450 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:05:43.872571 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:05:43.872774 2341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:05:43.873096 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:05:43.873136 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.873160 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:05:43.873178 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:05:43.881170 2341 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:36283
I20260925 10:05:43.881212 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:36283 every 8 connection(s)
I20260925 10:05:43.881628 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:05:43.882889 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.882475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:58283)
0925 10:05:43.882637 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.882728 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
W20260925 10:05:43.883378 2470 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:39743 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:39743: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
I20260925 10:05:43.883687 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.883586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:42547)
0925 10:05:43.883635 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.883666 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 10:05:43.884356 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:43.884256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:41687)
0925 10:05:43.884310 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:43.884336 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:05:43.884622 2470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:05:43.889190 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2341
I20260925 10:05:44.641352 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.641013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:58409)
0925 10:05:44.641221 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:44.641276 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260925 10:05:44.728361 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.728056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:58233)
0925 10:05:44.728236 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:44.728290 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260925 10:05:44.885329 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.885001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:40763)
0925 10:05:44.885195 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:44.885257 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260925 10:05:44.893828 2477 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:44.893976 2477 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:44.894622 2477 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869), 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
Sep 25 10:05:44 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.194@KRBTEST.COM
Sep 25 10:05:44 dist-test-slave-f9rl krb5kdc[900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330743, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.195@KRBTEST.COM
I20260925 10:05:44.909231 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.895222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36869 (local address 127.19.46.193:39013)
0925 10:05:44.895485 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:44.895500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:44.895583 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:44.896970 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:44.896972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:44.897375 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:44.897637 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:44.897645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.898841 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.898842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:44.899907 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:44.899914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.901295 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.901296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:44.901488 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:44.902468 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:44.902479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:44.904642 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:44.907267 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:44.907281 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:44.907284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:44.907546 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:44.907851 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:44.907852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:44.907853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:44.907976 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:44.908678 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:44.908685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:44.908814 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 10:05:44.908927 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:05:44.909120 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260925 10:05:44.909250 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.895415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:39013 (local address 127.19.46.194:36869)
0925 10:05:44.895587 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:44.895598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:44.895631 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:05:44.896634 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:44.896638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:44.896714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:44.896990 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:44.897781 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.898691 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.900113 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.901119 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.901322 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:44.904823 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:44.904848 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:44.904849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:44.904869 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:44.907163 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:44.907662 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:44.907666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:44.907667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:44.907768 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:44.908051 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:44.908052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:44.908052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:44.908330 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:44.908433 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:44.908985 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 10:05:44.909118 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
W20260925 10:05:44.910156 2091 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:44.910157 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.894823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36283 (local address 127.19.46.193:40991)
0925 10:05:44.894908 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:44.894923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:05:44.895011 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:05:44.896868 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:05:44.896873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:05:44.897365 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:05:44.897739 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:44.897744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.899386 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.899387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:44.900355 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:05:44.900359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.901501 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.901503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:05:44.901602 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:44.902602 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:05:44.902607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:05:44.905513 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:05:44.908412 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:44.908413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:44.908414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:44.908600 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:44.909027 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:05:44.909028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:05:44.909029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:05:44.909125 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:05:44.909743 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:05:44.909744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:05:44.909798 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:05:44.909909 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:05:44.910039 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:05:44.910282 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:44.895197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:40991 (local address 127.19.46.195:36283)
0925 10:05:44.895444 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:05:44.895458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:05:44.895656 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:05:44.896528 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:05:44.896532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:05:44.896598 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:05:44.896812 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:05:44.898014 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.899087 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.900601 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:05:44.901337 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:05:44.901566 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:05:44.905703 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:05:44.905709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:05:44.905710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:05:44.905725 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:05:44.908321 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:44.908712 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:44.908715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:44.908715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:44.908819 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:05:44.909211 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:05:44.909212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:05:44.909212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:05:44.909643 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:05:44.909750 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:05:44.909985 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:05:44.910146 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
W20260925 10:05:44.911155 2093 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:44.911239 2093 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f93b213465e4e5eabd42ee094201a47; no voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d
I20260925 10:05:44.911375 2477 raft_consensus.cc:2726] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:05:45.642463 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:45.642160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:50023)
0925 10:05:45.642360 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:45.642411 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260925 10:05:45.729600 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:45.729076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:50645)
0925 10:05:45.729493 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:45.729549 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":72,"threads_started":1}
I20260925 10:05:45.886675 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:45.886296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:50259)
0925 10:05:45.886539 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:45.886601 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
I20260925 10:05:46.643502 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:46.643226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:45283)
0925 10:05:46.643404 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:46.643452 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260925 10:05:46.730728 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:46.730389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:46059)
0925 10:05:46.730556 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:46.730671 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260925 10:05:46.887943 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:46.887559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:44639)
0925 10:05:46.887787 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:46.887888 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":138,"threads_started":1}
I20260925 10:05:47.152828 2487 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:47.152969 2487 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:47.153250 2487 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869), 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
W20260925 10:05:47.154134 2093 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
W20260925 10:05:47.154299 2091 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:47.154347 2091 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f93b213465e4e5eabd42ee094201a47; no voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d
I20260925 10:05:47.154474 2487 raft_consensus.cc:2726] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:05:47.644600 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:47.644256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:46709)
0925 10:05:47.644461 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:47.644498 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260925 10:05:47.731724 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:47.731426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:47337)
0925 10:05:47.731618 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:47.731667 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260925 10:05:47.889117 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:47.888793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:37417)
0925 10:05:47.889012 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:47.889061 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260925 10:05:48.645723 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:48.645356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:49565)
0925 10:05:48.645617 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:48.645666 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":143,"threads_started":1}
I20260925 10:05:48.732841 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:48.732495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:39861)
0925 10:05:48.732736 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:48.732786 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260925 10:05:48.890602 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:48.889983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:42307)
0925 10:05:48.890416 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:48.890528 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":143,"threads_started":1}
I20260925 10:05:49.646943 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:49.646592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:49359)
0925 10:05:49.646777 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:49.646894 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260925 10:05:49.733844 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:49.733571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:58501)
0925 10:05:49.733745 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:49.733794 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260925 10:05:49.891700 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:49.891257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:55487)
0925 10:05:49.891472 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:49.891546 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260925 10:05:50.248899 2497 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:50.249042 2497 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:50.249305 2497 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869), 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
W20260925 10:05:50.249814 2093 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
W20260925 10:05:50.249928 2091 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:50.249994 2091 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f93b213465e4e5eabd42ee094201a47; no voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d
I20260925 10:05:50.250285 2497 raft_consensus.cc:2726] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:05:50.648043 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:50.647747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:49153)
0925 10:05:50.647929 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:50.647985 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260925 10:05:50.735114 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:50.734787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:47223)
0925 10:05:50.735011 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:50.735061 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260925 10:05:50.892828 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:50.892524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:41775)
0925 10:05:50.892713 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:50.892771 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260925 10:05:51.649392 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:51.648975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:57091)
0925 10:05:51.649242 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:51.649284 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":139,"threads_started":1}
I20260925 10:05:51.736466 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:51.736136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:42585)
0925 10:05:51.736362 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:51.736405 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":177,"threads_started":1}
I20260925 10:05:51.894395 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:51.893997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:57821)
0925 10:05:51.894263 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:51.894338 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260925 10:05:52.650487 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:52.650141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:52419)
0925 10:05:52.650372 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:52.650437 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260925 10:05:52.738196 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:52.737832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:37331)
0925 10:05:52.738061 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:52.738123 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260925 10:05:52.895982 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:52.895583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:47907)
0925 10:05:52.895878 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:52.895930 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":241,"threads_started":1}
I20260925 10:05:53.651418 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:53.651157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:40053)
0925 10:05:53.651315 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:53.651356 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260925 10:05:53.739270 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:53.738961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:49099)
0925 10:05:53.739124 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:53.739208 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260925 10:05:53.896942 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:53.896576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:36137)
0925 10:05:53.896846 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:53.896891 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":183,"threads_started":1}
I20260925 10:05:54.652318 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:54.652044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:44181)
0925 10:05:54.652219 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:54.652269 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260925 10:05:54.740080 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:54.739844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:60987)
0925 10:05:54.739991 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:54.740029 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260925 10:05:54.897930 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:54.897653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:35969)
0925 10:05:54.897827 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:54.897877 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260925 10:05:55.183295 2513 raft_consensus.cc:482] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:05:55.183523 2513 raft_consensus.cc:504] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ce8fabbc3641f584ef31c501529443" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36869 } } peers { permanent_uuid: "9c9d166cdc494b75b1f992a148eac40d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36283 } } peers { permanent_uuid: "6f93b213465e4e5eabd42ee094201a47" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 39781 } }
I20260925 10:05:55.183840 2513 leader_election.cc:288] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869), 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283)
W20260925 10:05:55.184370 2091 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19ce8fabbc3641f584ef31c501529443 (127.19.46.194:36869): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
W20260925 10:05:55.184463 2093 leader_election.cc:341] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c9d166cdc494b75b1f992a148eac40d (127.19.46.195:36283): Not found: Tablet not found: b70f637e55e5496680c896bf8b74baeb
I20260925 10:05:55.184563 2093 leader_election.cc:302] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f93b213465e4e5eabd42ee094201a47; no voters: 19ce8fabbc3641f584ef31c501529443, 9c9d166cdc494b75b1f992a148eac40d
I20260925 10:05:55.184648 2513 raft_consensus.cc:2726] T b70f637e55e5496680c896bf8b74baeb P 6f93b213465e4e5eabd42ee094201a47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:05:55.653339 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:55.653028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:36047)
0925 10:05:55.653227 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:55.653289 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260925 10:05:55.741184 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:55.740837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:44297)
0925 10:05:55.741059 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:55.741130 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
I20260925 10:05:55.899192 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:55.898752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:58841)
0925 10:05:55.899015 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:55.899096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260925 10:05:56.654547 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:56.654119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:34815)
0925 10:05:56.654331 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:56.654496 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260925 10:05:56.742692 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:56.742163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:44509)
0925 10:05:56.742418 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:56.742497 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260925 10:05:56.900547 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:56.900118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:50761)
0925 10:05:56.900358 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:56.900475 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260925 10:05:57.655524 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:57.655291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:57577)
0925 10:05:57.655429 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:57.655476 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260925 10:05:57.743819 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:57.743489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:51483)
0925 10:05:57.743644 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:57.743701 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260925 10:05:57.901897 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:57.901627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:34293)
0925 10:05:57.901795 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:57.901845 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260925 10:05:58.656889 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:58.656518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.193:43633)
0925 10:05:58.656771 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:58.656828 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260925 10:05:58.744948 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:58.744641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.194:36031)
0925 10:05:58.744839 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:58.744898 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260925 10:05:58.900048 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2045
I20260925 10:05:58.902736 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:05:58.902537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:39743 (local address 127.19.46.195:46997)
0925 10:05:58.902660 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:05:58.902694 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260925 10:05:58.906229 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2208
I20260925 10:05:58.911900 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2341
2026-09-25T10:05:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18838 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:05:58 dist-test-slave-f9rl krb5kdc[2528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:05:58 dist-test-slave-f9rl krb5kdc[2528](info): set up 2 sockets
Sep 25 10:05:58 dist-test-slave-f9rl krb5kdc[2528](info): commencing operation
krb5kdc: starting...
W20260925 10:06:00.120242 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:06:00.282250 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:00.395915 2541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:00.396525 2541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:00.396616 2541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:00.399266 2541 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:06:00.399426 2541 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:06:00.399454 2541 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:06:00.399477 2541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:06:00.399505 2541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:00.399538 2541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:00.399564 2541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:06:00.399590 2541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:06:00.403262 2541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:00.403657 2541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:00.403935 2541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:00.410223 2549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:00.410223 2541 server_base.cc:1034] running on GCE node
W20260925 10:06:00.410336 2547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:00.410336 2546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:00.411356 2541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:00.411880 2541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:00.413132 2541 hybrid_clock.cc:648] HybridClock initialized: now 1790330760413117 us; error 46 us; skew 500 ppm
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:00.416937 2541 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:06:00.417619 2541 webserver.cc:460] Webserver started at http://127.19.46.254:41579/ using document root <none> and password file <none>
I20260925 10:06:00.417937 2541 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:00.418017 2541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:00.418329 2541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:00.420250 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "1ff1653703a440c5a627014efaab98ce"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "973e006ad7e0e3d2eaf31fed5ee5ba5e"
server_key_iv: "d4238dca758d4937235cb1a6f12fbafd"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.421034 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "1ff1653703a440c5a627014efaab98ce"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "973e006ad7e0e3d2eaf31fed5ee5ba5e"
server_key_iv: "d4238dca758d4937235cb1a6f12fbafd"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.422997 2541 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260925 10:06:00.424568 2556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.424980 2541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:06:00.425144 2541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "1ff1653703a440c5a627014efaab98ce"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "973e006ad7e0e3d2eaf31fed5ee5ba5e"
server_key_iv: "d4238dca758d4937235cb1a6f12fbafd"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.425292 2541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:00.441367 2541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:00.444693 2541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:00.444905 2541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:00.450937 2541 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:36713
I20260925 10:06:00.451013 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:36713 every 8 connection(s)
I20260925 10:06:00.451483 2541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:06:00.452576 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:00.455783 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap starting.
I20260925 10:06:00.456547 2609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:00.456925 2609 log.cc:824] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:00.458256 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: No bootstrap required, opened a new log
I20260925 10:06:00.459601 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2541
I20260925 10:06:00.459718 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:06:00.459949 19643 external_mini_cluster.cc:1428] Setting key bd142a40fdcac9f8c0d935c774cf9074
I20260925 10:06:00.461189 2609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:00.461390 2609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:00.461470 2609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Initialized, Role: FOLLOWER
I20260925 10:06:00.461673 2609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:00.461777 2609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:06:00.461826 2609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:06:00.461879 2609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:00.462884 2609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:00.463032 2609 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff1653703a440c5a627014efaab98ce; no voters:
I20260925 10:06:00.463254 2609 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:06:00.463289 2614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:06:00.463399 2614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 LEADER]: Becoming Leader. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Running, Role: LEADER
I20260925 10:06:00.463554 2609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:06:00.463542 2614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:00.464104 2615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:00.464193 2615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:00.464433 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff1653703a440c5a627014efaab98ce. Latest consensus state: current_term: 1 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:00.464548 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:00.464800 2622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:06:00.465353 2622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:06:00.467437 2622 catalog_manager.cc:1347] Generated new cluster ID: a06e8fccc0da471c8eeaf6d4039363ed
I20260925 10:06:00.467502 2622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:00.473721 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:00.460975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.19.46.254:36713)
0925 10:06:00.461196 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:00.461208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:00.461242 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:06:00.461349 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:00.461356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:00.461402 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:00.461595 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:00.462375 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.463912 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.465008 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.465847 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.465964 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:00.469076 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:00.469081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:00.469081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:00.469094 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:00.471357 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.471797 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.471801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.471801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.471938 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.472162 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.472163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.472164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.472557 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:00.472715 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:00.473310 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 10:06:00.473489 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":133,"threads_started":1}
I20260925 10:06:00.493427 2622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:06:00.494454 2622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:06:00.503803 2622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Generated new TSK 0
I20260925 10:06:00.504069 2622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:06:00.549929 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:00.656982 2635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:00.657281 2635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:00.657325 2635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:00.658964 2635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:00.659080 2635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:00.659140 2635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:06:00.661046 2635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:00.661403 2635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:00.661731 2635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:06:00.669829 2635 server_base.cc:1034] running on GCE node
W20260925 10:06:00.669752 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:00.669762 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:00.670011 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:00.670495 2635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:00.670810 2635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:00.672039 2635 hybrid_clock.cc:648] HybridClock initialized: now 1790330760671978 us; error 95 us; skew 500 ppm
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:00.676183 2635 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:06:00.677376 2635 webserver.cc:460] Webserver started at http://127.19.46.193:39575/ using document root <none> and password file <none>
I20260925 10:06:00.677838 2635 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:00.677932 2635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:00.678104 2635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:00.680095 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "7449d8cdfef14036977b1c3e672e50a8"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "14c263dbbc72ce563c77a32b1d987311"
server_key_iv: "d097b8dcb400818e401dbd8ec542a46f"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.680641 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "7449d8cdfef14036977b1c3e672e50a8"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "14c263dbbc72ce563c77a32b1d987311"
server_key_iv: "d097b8dcb400818e401dbd8ec542a46f"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.682768 2635 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260925 10:06:00.684139 2650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.684724 2635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:00.685105 2635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "7449d8cdfef14036977b1c3e672e50a8"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "14c263dbbc72ce563c77a32b1d987311"
server_key_iv: "d097b8dcb400818e401dbd8ec542a46f"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.685240 2635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:00.697140 2635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:00.701824 2635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:00.702067 2635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:00.702543 2635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:00.703194 2635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:00.703274 2635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.703320 2635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:00.703373 2635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.710840 2635 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:42549
I20260925 10:06:00.710875 2763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:42549 every 8 connection(s)
I20260925 10:06:00.711273 2635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:06:00.718933 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2635
I20260925 10:06:00.719061 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:06:00.719337 19643 external_mini_cluster.cc:1428] Setting key 3ee849f19658e47c165d890137b2593b
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:00.726326 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:00.712735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:47193 (local address 127.19.46.254:36713)
0925 10:06:00.712859 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:00.712861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:00.713135 (+ 274us) server_negotiation.cc:407] Connection header received
0925 10:06:00.714188 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:00.714191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:00.714214 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:00.714325 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:00.715608 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.716307 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.718091 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.718412 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.718509 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:00.721790 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:00.721792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:00.721792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:00.721802 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:00.724088 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.724715 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.724717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.724717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.724801 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.725380 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.725382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.725382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.725600 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:00.725693 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:00.726031 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 10:06:00.726141 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:06:00.726501 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:00.712547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:47193)
0925 10:06:00.712948 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:00.713005 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:00.713971 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:00.714420 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:00.714425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:00.715181 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:00.715452 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:00.715457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.716446 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.716447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:00.717900 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:00.717909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.718578 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.718579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:00.718752 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:00.719978 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:00.719990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:00.721681 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:00.724210 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:00.724221 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:00.724223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:00.724598 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:00.724928 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:00.724929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:00.724929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:00.725253 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:00.725689 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:00.725700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:00.725866 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 10:06:00.725990 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 10:06:00.726200 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20260925 10:06:00.727321 2764 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:00.727464 2764 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:00.727728 2764 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
I20260925 10:06:00.728500 2572 ts_manager.cc:194] Registered new tserver with Master: 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193:42549)
I20260925 10:06:00.729423 2572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:47193
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:06:00.759951 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:00.866290 2769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:00.866592 2769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:00.866627 2769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:00.868382 2769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:00.868465 2769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:00.868501 2769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:06:00.870900 2769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:00.871253 2769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:00.871565 2769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:00.877940 2777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:00.877928 2775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:00.877940 2774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:00.878247 2769 server_base.cc:1034] running on GCE node
I20260925 10:06:00.878854 2769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:00.879245 2769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:00.880468 2769 hybrid_clock.cc:648] HybridClock initialized: now 1790330760880427 us; error 72 us; skew 500 ppm
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:00.884223 2769 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:06:00.884768 2769 webserver.cc:460] Webserver started at http://127.19.46.194:44363/ using document root <none> and password file <none>
I20260925 10:06:00.885066 2769 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:00.885141 2769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:00.885301 2769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:00.886639 2769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "0c3fa6790a574f13b4631984034d3201"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "209a1c00a85d4bab24d56bd27bdf1df4"
server_key_iv: "ca79c7210b711e9abb579c65c7b99122"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.887079 2769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "0c3fa6790a574f13b4631984034d3201"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "209a1c00a85d4bab24d56bd27bdf1df4"
server_key_iv: "ca79c7210b711e9abb579c65c7b99122"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.889189 2769 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:06:00.890683 2784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.891216 2769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:06:00.891645 2769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "0c3fa6790a574f13b4631984034d3201"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "209a1c00a85d4bab24d56bd27bdf1df4"
server_key_iv: "ca79c7210b711e9abb579c65c7b99122"
server_key_version: "encryptionkey@0"
I20260925 10:06:00.891950 2769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:00.924360 2769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:00.927057 2769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:00.927249 2769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:00.927526 2769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:00.927925 2769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:00.927983 2769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.928027 2769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:00.928071 2769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:00.936995 2769 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34949
I20260925 10:06:00.937305 2898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34949 every 8 connection(s)
I20260925 10:06:00.937562 2769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:06:00.938022 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2769
I20260925 10:06:00.938170 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:06:00.938395 19643 external_mini_cluster.cc:1428] Setting key 0ab0362a827761810eff41f851f537de
Sep 25 10:06:00 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330760, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:06:00.951162 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:00.938571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:34371 (local address 127.19.46.254:36713)
0925 10:06:00.938633 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:00.938636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:00.938956 (+ 320us) server_negotiation.cc:407] Connection header received
0925 10:06:00.939549 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:00.939550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:00.939564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:00.939656 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:00.940888 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.941658 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.943057 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.943288 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.943495 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:00.946958 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:00.946961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:00.946961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:00.946972 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:00.949003 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.949664 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.949666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.949666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.949737 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:00.950048 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:00.950049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:00.950049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:00.950201 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:00.950301 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:00.950839 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 10:06:00.951004 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:06:00.951611 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:00.938567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:34371)
0925 10:06:00.938813 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:00.938858 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:00.939441 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:00.939808 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:00.939812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:00.940499 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:00.940739 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:00.940744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.941772 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.941773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:00.942921 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:00.942924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:00.943505 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:00.943507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:00.943682 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:00.944386 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:00.944397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:00.946811 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:00.949134 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:00.949149 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:00.949153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:00.949560 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:00.949825 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:00.949826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:00.949827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:00.949964 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:00.950304 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:00.950314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:00.950499 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 10:06:00.950790 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:06:00.951180 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260925 10:06:00.952852 2899 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:00.952983 2899 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:00.953269 2899 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
I20260925 10:06:00.953948 2572 ts_manager.cc:194] Registered new tserver with Master: 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949)
I20260925 10:06:00.954545 2572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:34371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:06:00.979264 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:01.097132 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:01.097559 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:01.097620 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:01.099618 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:01.099692 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:01.099769 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:06:01.101994 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:01.102396 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:01.102821 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:01.109702 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:01.109628 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:01.110062 2904 server_base.cc:1034] running on GCE node
W20260925 10:06:01.110086 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:01.110493 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:01.110831 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:01.112035 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790330761111980 us; error 84 us; skew 500 ppm
Sep 25 10:06:01 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330761, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:01.121680 2904 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:06:01.122958 2904 webserver.cc:460] Webserver started at http://127.19.46.195:46401/ using document root <none> and password file <none>
I20260925 10:06:01.123286 2904 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:01.123370 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:01.123538 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:01.124775 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
format_stamp: "Formatted at 2026-09-25 10:06:01 on dist-test-slave-f9rl"
server_key: "e2435a03ef74cf74204669b4912b0cd0"
server_key_iv: "93732961c7f0a3237f69c74929c959c9"
server_key_version: "encryptionkey@0"
I20260925 10:06:01.125173 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
format_stamp: "Formatted at 2026-09-25 10:06:01 on dist-test-slave-f9rl"
server_key: "e2435a03ef74cf74204669b4912b0cd0"
server_key_iv: "93732961c7f0a3237f69c74929c959c9"
server_key_version: "encryptionkey@0"
I20260925 10:06:01.127252 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:06:01.128459 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:01.128940 2904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:01.129057 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
format_stamp: "Formatted at 2026-09-25 10:06:01 on dist-test-slave-f9rl"
server_key: "e2435a03ef74cf74204669b4912b0cd0"
server_key_iv: "93732961c7f0a3237f69c74929c959c9"
server_key_version: "encryptionkey@0"
I20260925 10:06:01.129186 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:01.162094 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:01.165834 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:01.166046 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:01.166385 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:01.166776 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:01.166841 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:01.166889 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:01.166920 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:01.175988 2904 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:43475
I20260925 10:06:01.176126 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:43475 every 8 connection(s)
I20260925 10:06:01.176549 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:06:01.178251 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 2904
I20260925 10:06:01.178395 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:06:01.178653 19643 external_mini_cluster.cc:1428] Setting key c8697029c55ee55e0a6c439ebb0126fa
Sep 25 10:06:01 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330761, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:01.188442 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.177554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:45749 (local address 127.19.46.254:36713)
0925 10:06:01.177630 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.177632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.177918 (+ 286us) server_negotiation.cc:407] Connection header received
0925 10:06:01.178569 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.178571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.178591 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.178667 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:01.179975 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.180595 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.181822 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.182053 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.182152 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.184897 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:01.184899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:01.184899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:01.184908 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:01.186604 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:01.187208 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:01.187210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:01.187210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:01.187282 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:01.187597 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:01.187598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:01.187599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:01.187738 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:01.187829 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.188203 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.188334 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:06:01.188767 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.177536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:45749)
0925 10:06:01.177776 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.177818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.178434 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.178732 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.178736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.179635 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:01.179870 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.179875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.180717 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.180718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.181724 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.181726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.182167 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.182168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.182315 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.182958 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:01.182967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:01.184768 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:01.186811 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:01.186825 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:01.186828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:01.187113 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:01.187369 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:01.187370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:01.187370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:01.187496 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:01.187820 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:01.187827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:01.188011 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.188151 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.188347 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 10:06:01.189640 3033 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:01.189741 3033 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:01.189963 3033 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
I20260925 10:06:01.190460 2572 ts_manager.cc:194] Registered new tserver with Master: 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
I20260925 10:06:01.191033 2572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:45749
I20260925 10:06:01.191051 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:06:01.201228 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.192502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.19.46.254:36713)
0925 10:06:01.192583 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.192585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.192722 (+ 137us) server_negotiation.cc:407] Connection header received
0925 10:06:01.192769 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.192769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.192779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.192834 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:01.193613 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.194230 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.195308 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.195610 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.195790 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.197502 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:01.197504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:01.197504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:01.197511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:01.199341 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:01.199863 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:01.199865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:01.199866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:01.199966 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:01.200358 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:01.200360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:01.200360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:01.200566 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:01.200723 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.201021 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.201096 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:06:01.203608 2572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:06:01.205411 2572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:06:01.213382 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42549 (local address 127.0.0.1:51328)
0925 10:06:01.208682 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.208695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.208753 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.208971 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.208975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.208977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:01.209093 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.209096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210473 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.210474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.211826 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.211829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.213060 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.213062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.213123 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.213127 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.213211 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.213309 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":48,"thread_start_us":78,"threads_started":1}
I20260925 10:06:01.213379 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.19.46.193:42549)
0925 10:06:01.208800 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.208804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.208812 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:06:01.208849 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.208851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.208863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.208923 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:01.209240 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210354 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.212073 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.212908 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.212976 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.213026 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.213230 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.213281 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":60,"threads_started":1}
I20260925 10:06:01.213488 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.19.46.195:43475)
0925 10:06:01.208814 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.208817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.208828 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:06:01.208867 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.208869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.208892 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.208994 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:01.209201 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210717 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.211822 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.212683 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.212770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.212937 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.213278 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.213356 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":92,"threads_started":1}
I20260925 10:06:01.213847 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:43475 (local address 127.0.0.1:45052)
0925 10:06:01.208541 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.208585 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.208662 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.209013 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.209014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.209016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:01.209101 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.209104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210830 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.210831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.211719 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.211732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.212849 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.212852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.213081 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.213104 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.213209 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.213760 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":520,"thread_start_us":57,"threads_started":1}
I20260925 10:06:01.214042 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.19.46.194:34949)
0925 10:06:01.208723 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.208728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.209159 (+ 431us) server_negotiation.cc:407] Connection header received
0925 10:06:01.209214 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.209216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.209234 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.209332 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:01.209594 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210697 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.212087 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.213428 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.213545 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.213612 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.213890 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.213957 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":89,"threads_started":1}
I20260925 10:06:01.214304 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.208266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34949 (local address 127.0.0.1:53574)
0925 10:06:01.209072 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.209083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.209140 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.209353 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.209354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.209356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:01.209506 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.209509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.210808 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.210809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.212016 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.212020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.213656 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.213657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.213731 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.213735 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.213792 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.213833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":277,"thread_start_us":69,"threads_started":1}
I20260925 10:06:01.214838 2967 tablet_service.cc:1467] Processing CreateTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d (DEFAULT_TABLE table=test-table [id=3f518fa21f6b40df8508342da36c0f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:01.214844 2698 tablet_service.cc:1467] Processing CreateTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d (DEFAULT_TABLE table=test-table [id=3f518fa21f6b40df8508342da36c0f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:01.215207 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f24c441a43e43ae823a7ed6b787a93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:01.215225 2698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f24c441a43e43ae823a7ed6b787a93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:01.218421 3050 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Bootstrap starting.
I20260925 10:06:01.219108 2833 tablet_service.cc:1467] Processing CreateTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d (DEFAULT_TABLE table=test-table [id=3f518fa21f6b40df8508342da36c0f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:01.219290 3050 tablet_bootstrap.cc:654] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:01.219488 2833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f24c441a43e43ae823a7ed6b787a93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:01.219907 3050 log.cc:824] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:01.220328 3051 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap starting.
I20260925 10:06:01.220920 3050 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: No bootstrap required, opened a new log
I20260925 10:06:01.221027 3050 ts_tablet_manager.cc:1397] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:06:01.221074 3051 tablet_bootstrap.cc:654] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:01.221778 3051 log.cc:824] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:01.222847 3051 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: No bootstrap required, opened a new log
I20260925 10:06:01.222932 3051 ts_tablet_manager.cc:1397] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:06:01.222831 3050 raft_consensus.cc:348] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.223016 3050 raft_consensus.cc:374] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:01.223078 3050 raft_consensus.cc:729] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470b6e548a434c94b5d6bb1d87cc0e03, State: Initialized, Role: FOLLOWER
I20260925 10:06:01.223237 3050 consensus_queue.cc:260] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.223553 3050 ts_tablet_manager.cc:1428] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:06:01.223608 3055 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Bootstrap starting.
I20260925 10:06:01.223749 3033 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
I20260925 10:06:01.224416 3055 tablet_bootstrap.cc:654] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:01.225492 3051 raft_consensus.cc:348] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.225687 3051 raft_consensus.cc:374] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:01.225718 3051 raft_consensus.cc:729] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Initialized, Role: FOLLOWER
I20260925 10:06:01.225739 3055 log.cc:824] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:01.225883 3051 consensus_queue.cc:260] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.226263 2764 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
I20260925 10:06:01.226245 3051 ts_tablet_manager.cc:1428] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:06:01.226634 3055 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: No bootstrap required, opened a new log
I20260925 10:06:01.226758 3055 ts_tablet_manager.cc:1397] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:06:01.228513 3055 raft_consensus.cc:348] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.228659 3055 raft_consensus.cc:374] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:01.228683 3055 raft_consensus.cc:729] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3fa6790a574f13b4631984034d3201, State: Initialized, Role: FOLLOWER
I20260925 10:06:01.228776 3055 consensus_queue.cc:260] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.229084 3055 ts_tablet_manager.cc:1428] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:06:01.229102 2899 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
I20260925 10:06:01.417095 3057 raft_consensus.cc:482] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:01.417389 3057 raft_consensus.cc:504] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.417928 3057 leader_election.cc:288] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475), 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949)
I20260925 10:06:01.423677 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.418248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:53703 (local address 127.19.46.195:43475)
0925 10:06:01.418431 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.418433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.418445 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:06:01.418479 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.418480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.418490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.418547 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:01.419865 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.420980 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.422384 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.422884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.422955 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.422963 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.423424 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.423475 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260925 10:06:01.423799 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.418150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:43475 (local address 127.19.46.193:53703)
0925 10:06:01.418331 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.418347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.418413 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.419344 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.419346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.419350 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:01.419456 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.419458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.421144 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.421146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.422271 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.422274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.422989 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.422990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.423077 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.423083 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.423337 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.423392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260925 10:06:01.423899 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.419292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:42167 (local address 127.19.46.194:34949)
0925 10:06:01.419361 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.419363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.420038 (+ 675us) server_negotiation.cc:407] Connection header received
0925 10:06:01.420089 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.420090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.420110 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.420352 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:01.420629 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.421231 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.422553 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.423312 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.423383 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.423392 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.423697 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.423796 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:06:01.423813 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.418357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34949 (local address 127.19.46.193:42167)
0925 10:06:01.419874 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.419891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:01.419982 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:01.420367 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:01.420369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:01.420371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:01.420461 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.420463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.421366 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.421367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.422435 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:01.422438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.423427 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.423429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:01.423505 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.423510 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:06:01.423581 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:06:01.423624 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1486,"mutex_wait_us":1,"thread_start_us":1093,"threads_started":1}
I20260925 10:06:01.424955 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f24c441a43e43ae823a7ed6b787a93d" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" is_pre_election: true
I20260925 10:06:01.424978 2853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f24c441a43e43ae823a7ed6b787a93d" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3fa6790a574f13b4631984034d3201" is_pre_election: true
I20260925 10:06:01.425158 2987 raft_consensus.cc:2454] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 0.
I20260925 10:06:01.425175 2853 raft_consensus.cc:2454] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 0.
I20260925 10:06:01.425513 2653 leader_election.cc:302] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 470b6e548a434c94b5d6bb1d87cc0e03, 7449d8cdfef14036977b1c3e672e50a8; no voters:
I20260925 10:06:01.425820 3057 raft_consensus.cc:2781] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:06:01.425896 3057 raft_consensus.cc:482] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:06:01.425925 3057 raft_consensus.cc:3037] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:01.426944 3057 raft_consensus.cc:504] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.427147 3057 leader_election.cc:288] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 election: Requested vote from peers 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475), 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949)
I20260925 10:06:01.427359 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f24c441a43e43ae823a7ed6b787a93d" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
I20260925 10:06:01.427451 2987 raft_consensus.cc:3037] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:06:01.427464 3034 tablet.cc:2369] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:06:01.427474 2853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f24c441a43e43ae823a7ed6b787a93d" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3fa6790a574f13b4631984034d3201"
I20260925 10:06:01.427605 2853 raft_consensus.cc:3037] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:01.428383 2987 raft_consensus.cc:2454] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 1.
I20260925 10:06:01.428594 2653 leader_election.cc:302] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 470b6e548a434c94b5d6bb1d87cc0e03, 7449d8cdfef14036977b1c3e672e50a8; no voters:
I20260925 10:06:01.428716 3057 raft_consensus.cc:2781] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:06:01.428758 2853 raft_consensus.cc:2454] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 1.
I20260925 10:06:01.428932 3057 raft_consensus.cc:686] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Running, Role: LEADER
I20260925 10:06:01.429047 3057 consensus_queue.cc:237] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:01.431504 2573 catalog_manager.cc:5495] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "7449d8cdfef14036977b1c3e672e50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } health_report { overall_health: UNKNOWN } } }
W20260925 10:06:01.438656 2900 tablet.cc:2369] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:06:01.448596 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.444767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.19.46.193:42549)
0925 10:06:01.444832 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:01.444834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:01.444845 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:06:01.444897 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:01.444898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:01.444918 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:01.444988 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:06:01.445228 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.446286 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.447555 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:01.447836 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:01.447915 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:01.448137 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:06:01.448263 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:06:01.448343 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:01.448421 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:06:01.448444 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:06:01.450790 2987 raft_consensus.cc:1261] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Refusing update from remote peer 7449d8cdfef14036977b1c3e672e50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:06:01.450861 2853 raft_consensus.cc:1261] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Refusing update from remote peer 7449d8cdfef14036977b1c3e672e50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:06:01.451198 3057 consensus_queue.cc:1035] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:06:01.451359 3061 consensus_queue.cc:1035] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:06:01.453961 3064 mvcc.cc:205] Tried to move back new op lower bound from 7333194798899122176 to 7333194798815805440. Current Snapshot: MvccSnapshot[applied={T|T < 7333194798899122176}]
I20260925 10:06:01.454897 3066 mvcc.cc:205] Tried to move back new op lower bound from 7333194798899122176 to 7333194798815805440. Current Snapshot: MvccSnapshot[applied={T|T < 7333194798899122176}]
I20260925 10:06:01.455286 3065 mvcc.cc:205] Tried to move back new op lower bound from 7333194798899122176 to 7333194798815805440. Current Snapshot: MvccSnapshot[applied={T|T < 7333194798899122176}]
I20260925 10:06:01.457008 2573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:06:01.457121 2573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:06:01.458384 2573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Sending DeleteTablet for 3 replicas of tablet 7f24c441a43e43ae823a7ed6b787a93d
I20260925 10:06:01.459067 2967 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:01 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:45052
I20260925 10:06:01.459132 2833 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:01 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:53574
I20260925 10:06:01.459652 3075 tablet_replica.cc:330] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: stopping tablet replica
I20260925 10:06:01.459846 2698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:01 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:51328
I20260925 10:06:01.459903 3075 raft_consensus.cc:2229] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:06:01.460026 3075 raft_consensus.cc:2258] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:01.460286 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2635
I20260925 10:06:01.462633 3075 ts_tablet_manager.cc:1905] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:01.462781 3074 tablet_replica.cc:330] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: stopping tablet replica
I20260925 10:06:01.463413 3074 raft_consensus.cc:2229] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:06:01.463919 3074 raft_consensus.cc:2258] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:01.465158 3074 ts_tablet_manager.cc:1905] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:01.466859 3075 ts_tablet_manager.cc:1918] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:06:01.466965 3075 log.cc:1196] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/7f24c441a43e43ae823a7ed6b787a93d
I20260925 10:06:01.467303 3075 ts_tablet_manager.cc:1939] T 7f24c441a43e43ae823a7ed6b787a93d P 0c3fa6790a574f13b4631984034d3201: Deleting consensus metadata
I20260925 10:06:01.467286 3074 ts_tablet_manager.cc:1918] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:06:01.467371 3074 log.cc:1196] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/7f24c441a43e43ae823a7ed6b787a93d
I20260925 10:06:01.467728 3074 ts_tablet_manager.cc:1939] T 7f24c441a43e43ae823a7ed6b787a93d P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting consensus metadata
I20260925 10:06:01.467917 2560 catalog_manager.cc:4841] TS 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): tablet 7f24c441a43e43ae823a7ed6b787a93d (table test-table [id=3f518fa21f6b40df8508342da36c0f5e]) successfully deleted
I20260925 10:06:01.468257 2559 catalog_manager.cc:4841] TS 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): tablet 7f24c441a43e43ae823a7ed6b787a93d (table test-table [id=3f518fa21f6b40df8508342da36c0f5e]) successfully deleted
W20260925 10:06:01.468521 2558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:42549 (error 108)
I20260925 10:06:01.468822 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2769
I20260925 10:06:01.469655 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:01.469018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42549 (local address 127.0.0.1:51344)
0925 10:06:01.469554 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:01.469600 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509}
W20260925 10:06:01.471302 2558 catalog_manager.cc:4584] TS 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193:42549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f24c441a43e43ae823a7ed6b787a93d: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42549: connect: Connection refused (error 111)
I20260925 10:06:01.475124 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2904
I20260925 10:06:01.482053 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 2541
I20260925 10:06:21.489711 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--webserver_interface=127.19.46.254
--webserver_port=41579
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:21.576699 3078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:21.576941 3078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:21.576964 3078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:21.578239 3078 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:06:21.578294 3078 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:06:21.578308 3078 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:06:21.578320 3078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:06:21.578334 3078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:21.578348 3078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:21.578394 3078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:06:21.578411 3078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:06:21.580039 3078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:21.580312 3078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:21.580557 3078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:21.590474 3087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:21.590430 3084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:21.590458 3078 server_base.cc:1034] running on GCE node
W20260925 10:06:21.590456 3085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:21.591234 3078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:21.591765 3078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:21.593035 3078 hybrid_clock.cc:648] HybridClock initialized: now 1790330781592938 us; error 62 us; skew 500 ppm
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:21.596372 3078 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:06:21.597045 3078 webserver.cc:460] Webserver started at http://127.19.46.254:41579/ using document root <none> and password file <none>
I20260925 10:06:21.597344 3078 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:21.597438 3078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:21.599241 3078 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:21.600049 3094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:21.600359 3078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:21.600494 3078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "1ff1653703a440c5a627014efaab98ce"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "973e006ad7e0e3d2eaf31fed5ee5ba5e"
server_key_iv: "d4238dca758d4937235cb1a6f12fbafd"
server_key_version: "encryptionkey@0"
I20260925 10:06:21.601040 3078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:21.611598 3078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:21.614836 3078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:21.615149 3078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:21.620590 3078 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:36713
I20260925 10:06:21.620663 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:36713 every 8 connection(s)
I20260925 10:06:21.621346 3078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:06:21.623510 3147 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:06:21.624276 3147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap starting.
I20260925 10:06:21.625685 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3078
I20260925 10:06:21.625982 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:42549
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=39575
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:06:21.633416 3147 log.cc:824] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:21.636111 3147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:06:21.636941 3147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap complete.
I20260925 10:06:21.639858 3147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:21.640121 3147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Initialized, Role: FOLLOWER
I20260925 10:06:21.640300 3147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:21.640372 3147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:06:21.640457 3147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:06:21.640508 3147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:06:21.641764 3147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:21.641868 3147 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff1653703a440c5a627014efaab98ce; no voters:
I20260925 10:06:21.642081 3147 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:06:21.642125 3151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:06:21.642242 3151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 LEADER]: Becoming Leader. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Running, Role: LEADER
I20260925 10:06:21.642388 3147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:06:21.642365 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:21.642894 3153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff1653703a440c5a627014efaab98ce. Latest consensus state: current_term: 2 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:21.642994 3153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:21.643348 3159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:06:21.643396 3152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:21.643479 3152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:21.644191 3159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:06:21.644297 3159 catalog_manager.cc:1259] Loaded cluster ID: a06e8fccc0da471c8eeaf6d4039363ed
I20260925 10:06:21.644330 3159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:06:21.645656 3159 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:06:21.645849 3159 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Loaded TSK: 0
I20260925 10:06:21.661726 3159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Generated new TSK 1
I20260925 10:06:21.661886 3159 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:06:21.723963 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:21.724215 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:21.724255 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:21.725629 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:21.725696 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:21.725747 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:06:21.727309 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=39575
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:21.727598 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:21.727849 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:21.733256 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:21.733294 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:21.733251 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:21.733847 3149 server_base.cc:1034] running on GCE node
I20260925 10:06:21.734141 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:21.734455 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:21.735682 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790330781735658 us; error 40 us; skew 500 ppm
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:21.738479 3149 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:06:21.739066 3149 webserver.cc:460] Webserver started at http://127.19.46.193:39575/ using document root <none> and password file <none>
I20260925 10:06:21.739310 3149 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:21.739390 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:21.741878 3149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:06:21.742820 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:21.743032 3149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:21.743130 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "7449d8cdfef14036977b1c3e672e50a8"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "14c263dbbc72ce563c77a32b1d987311"
server_key_iv: "d097b8dcb400818e401dbd8ec542a46f"
server_key_version: "encryptionkey@0"
I20260925 10:06:21.743577 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:21.762101 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:21.764822 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:21.764988 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:21.765295 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:21.765869 3191 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:06:21.767519 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:06:21.767588 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:06:21.767620 3149 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:06:21.768363 3149 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:06:21.768412 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:06:21.768471 3191 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap starting.
I20260925 10:06:21.777671 3149 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:42549
I20260925 10:06:21.777942 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:42549 every 8 connection(s)
I20260925 10:06:21.778290 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:06:21.781992 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3149
I20260925 10:06:21.782373 3191 log.cc:824] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:21.782480 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:34949
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=44363
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:06:21.786427 3191 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:06:21.787216 3191 tablet_bootstrap.cc:492] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap complete.
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:21.788650 3191 ts_tablet_manager.cc:1397] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20260925 10:06:21.791280 3191 raft_consensus.cc:348] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:21.791710 3191 raft_consensus.cc:729] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Initialized, Role: FOLLOWER
I20260925 10:06:21.792068 3191 consensus_queue.cc:260] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } } peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } }
I20260925 10:06:21.792467 3191 ts_tablet_manager.cc:1428] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 10:06:21.794100 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.779257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:45347 (local address 127.19.46.254:36713)
0925 10:06:21.779510 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:21.779519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:21.779566 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:06:21.780496 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:21.780503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:21.780561 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:21.780715 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:21.781818 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.783724 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.785329 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.785956 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.786073 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.789295 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:21.789301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:21.789301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:21.789316 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:21.791884 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.792493 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.792498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.792498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.792638 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.792927 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.792928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.792929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.793209 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:21.793313 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:21.793738 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 10:06:21.793895 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260925 10:06:21.794108 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.779186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:45347)
0925 10:06:21.779401 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:21.779463 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:21.780229 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:21.780772 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:21.780777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:21.781284 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:21.781646 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.781655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.783911 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.783913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.785205 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.785209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.786110 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.786112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.786295 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.786998 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:21.787011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:21.789129 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:21.792003 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.792009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.792012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.792377 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.792737 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.792738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.792738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.792840 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.793304 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:21.793311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:21.793521 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 10:06:21.793660 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:06:21.793875 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260925 10:06:21.794817 3299 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:21.794961 3299 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:21.795157 3299 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
I20260925 10:06:21.795964 3111 ts_manager.cc:194] Registered new tserver with Master: 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193:42549)
I20260925 10:06:21.797223 3111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:45347
I20260925 10:06:21.798467 3299 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.193@KRBTEST.COM
I20260925 10:06:21.809351 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.796649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42549 (local address 127.0.0.1:49090)
0925 10:06:21.797651 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:21.797691 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:21.797761 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:21.798121 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:21.798128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:21.798527 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:21.798600 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.798603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.799936 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.799937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.800761 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.800764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.801512 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.801514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.801624 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.802670 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:21.802677 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:21.804717 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:21.807381 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.807390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.807392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.807642 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.808025 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.808027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.808028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.808162 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.808729 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:21.808747 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:21.808994 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 10:06:21.809108 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:06:21.809265 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":109,"threads_started":1}
I20260925 10:06:21.809401 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.796932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.19.46.193:42549)
0925 10:06:21.797907 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:21.797912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:21.797921 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:06:21.797958 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:21.797960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:21.798007 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:21.798085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:21.798712 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.799832 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.800858 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.801350 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.801426 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.804860 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:21.804874 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:21.804874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:21.804883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:21.807154 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.807737 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.807741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.807742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.807826 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.808334 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.808336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.808336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.808606 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:21.808728 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:21.809170 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 10:06:21.809282 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":79,"threads_started":1}
I20260925 10:06:21.810281 3233 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f24c441a43e43ae823a7ed6b787a93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:01 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49090
I20260925 10:06:21.811053 3308 tablet_replica.cc:330] stopping tablet replica
I20260925 10:06:21.811177 3308 raft_consensus.cc:2229] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:06:21.811271 3308 raft_consensus.cc:2258] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:21.811586 3308 ts_tablet_manager.cc:1905] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:21.814000 3308 ts_tablet_manager.cc:1918] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:06:21.814081 3308 log.cc:1196] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal/wals/7f24c441a43e43ae823a7ed6b787a93d
I20260925 10:06:21.814343 3308 ts_tablet_manager.cc:1939] T 7f24c441a43e43ae823a7ed6b787a93d P 7449d8cdfef14036977b1c3e672e50a8: Deleting consensus metadata
I20260925 10:06:21.815522 3096 catalog_manager.cc:4841] TS 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193:42549): tablet 7f24c441a43e43ae823a7ed6b787a93d (table test-table [id=3f518fa21f6b40df8508342da36c0f5e]) successfully deleted
W20260925 10:06:21.897301 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:21.897634 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:21.897680 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:21.899290 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:21.899370 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:21.899418 3304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:06:21.901089 3304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:34949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=44363
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:21.901398 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:21.901690 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:21.908218 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:21.908411 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:21.908473 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:21.909780 3304 server_base.cc:1034] running on GCE node
I20260925 10:06:21.910117 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:21.910434 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:21.911659 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790330781911611 us; error 62 us; skew 500 ppm
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:21.916450 3304 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:06:21.917043 3304 webserver.cc:460] Webserver started at http://127.19.46.194:44363/ using document root <none> and password file <none>
I20260925 10:06:21.917292 3304 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:21.917379 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:21.919086 3304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:21.920051 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:21.920373 3304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:21.920506 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "0c3fa6790a574f13b4631984034d3201"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "209a1c00a85d4bab24d56bd27bdf1df4"
server_key_iv: "ca79c7210b711e9abb579c65c7b99122"
server_key_version: "encryptionkey@0"
I20260925 10:06:21.920969 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:21.943109 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:21.947036 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:21.947290 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:21.947551 3304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:21.948130 3304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:21.948199 3304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:21.948241 3304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:21.948276 3304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:21.957332 3304 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34949
I20260925 10:06:21.957894 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:06:21.958045 3436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34949 every 8 connection(s)
I20260925 10:06:21.962061 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3304
I20260925 10:06:21.962637 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:43475
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=46401
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:06:21 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330781, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:21.970263 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.958890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:36021 (local address 127.19.46.254:36713)
0925 10:06:21.959015 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:21.959018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:21.959237 (+ 219us) server_negotiation.cc:407] Connection header received
0925 10:06:21.959920 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:21.959921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:21.959940 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:21.960010 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:21.961136 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.961795 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.963093 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.963329 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.963428 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.966555 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:21.966557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:21.966557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:21.966574 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:21.968498 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.969021 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.969022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.969023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.969157 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:21.969473 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:21.969474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:21.969474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:21.969617 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:21.969698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:21.970020 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:06:21.970158 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:06:21.970479 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:21.958788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:36021)
0925 10:06:21.959070 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:21.959134 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:21.959787 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:21.960095 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:21.960099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:21.960779 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:21.961018 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.961023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.961924 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.961925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.962990 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:21.962994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:21.963452 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:21.963453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:21.963599 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:21.964392 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:21.964406 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:21.966418 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:21.968636 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.968650 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.968653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.968931 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.969237 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:21.969238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:21.969238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:21.969364 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:21.969704 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:21.969712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:21.969870 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:06:21.969998 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:06:21.970194 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260925 10:06:21.971066 3437 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:21.971223 3437 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:21.971480 3437 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
I20260925 10:06:21.971971 3111 ts_manager.cc:194] Registered new tserver with Master: 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949)
I20260925 10:06:21.972393 3111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:36021
W20260925 10:06:22.059576 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:22.059844 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:22.059886 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:22.061153 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:22.061233 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:22.061281 3440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:06:22.062904 3440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:43475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=46401
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:22.063210 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:22.063453 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:22.068876 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:22.068864 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:22.068949 3440 server_base.cc:1034] running on GCE node
W20260925 10:06:22.068864 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:22.069321 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:22.069593 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:22.070771 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790330782070752 us; error 46 us; skew 500 ppm
Sep 25 10:06:22 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330782, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:22.073611 3440 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:06:22.074100 3440 webserver.cc:460] Webserver started at http://127.19.46.195:46401/ using document root <none> and password file <none>
I20260925 10:06:22.074318 3440 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:22.074379 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:22.076038 3440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:06:22.076761 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:22.076958 3440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:22.077060 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
format_stamp: "Formatted at 2026-09-25 10:06:01 on dist-test-slave-f9rl"
server_key: "e2435a03ef74cf74204669b4912b0cd0"
server_key_iv: "93732961c7f0a3237f69c74929c959c9"
server_key_version: "encryptionkey@0"
I20260925 10:06:22.077481 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:22.087533 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:22.090309 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:22.090435 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:22.090632 3440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:22.090920 3440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:22.090952 3440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:22.090974 3440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:22.090987 3440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:22.098413 3440 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:43475
I20260925 10:06:22.098477 3568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:43475 every 8 connection(s)
I20260925 10:06:22.098865 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:06:22.099843 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3440
Sep 25 10:06:22 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330782, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:22.109007 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.100331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.19.46.254:36713)
0925 10:06:22.100456 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.100458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.100508 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:06:22.100609 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.100610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.100622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.100753 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:22.101886 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.102643 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.103849 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.104096 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.104154 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.105826 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:22.105829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:22.105829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:22.105837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:22.107549 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.107973 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.107975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.107975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.108050 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.108288 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.108289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.108289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.108471 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:22.108716 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.108754 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.108879 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260925 10:06:22.111059 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.099699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:40943 (local address 127.19.46.254:36713)
0925 10:06:22.099775 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.099777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.099997 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:06:22.100686 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.100687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.100696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.100751 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:22.102250 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.102928 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.104329 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.104642 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.104731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.107498 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:22.107500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:22.107500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:22.107516 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:22.109389 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.109981 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.109982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.109983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.110045 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.110357 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.110358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.110359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.110484 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:22.110562 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.110863 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.110953 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260925 10:06:22.111307 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.099680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:40943)
0925 10:06:22.099862 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.099910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:22.100608 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:22.100834 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:22.100838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:22.101919 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:22.102142 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.102149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.103035 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.103036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.104233 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.104235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.104766 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.104768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.104968 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.105563 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:22.105572 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:22.107364 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:22.109509 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:22.109519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:22.109522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:22.109891 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:22.110122 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:22.110123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:22.110124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:22.110277 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:22.110552 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:22.110559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:22.110720 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 10:06:22.110841 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:06:22.111028 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 10:06:22.111865 3569 heartbeater.cc:347] Connected to a master server at 127.19.46.254:36713
I20260925 10:06:22.111979 3569 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:22.112200 3569 heartbeater.cc:511] Master 127.19.46.254:36713 requested a full tablet report, sending...
I20260925 10:06:22.112615 3111 ts_manager.cc:194] Registered new tserver with Master: 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
I20260925 10:06:22.113022 3111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:40943
I20260925 10:06:22.121519 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:06:22.125572 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.122349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.19.46.254:36713)
0925 10:06:22.122581 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.122584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.122730 (+ 146us) server_negotiation.cc:407] Connection header received
0925 10:06:22.122862 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.122863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.122905 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.122991 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:06:22.123155 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.123693 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.124615 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.124881 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.124968 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.125175 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:06:22.125235 (+ 60us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:06:22.125479 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34952: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 10:06:22.125645 3302 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34952: token expired
W20260925 10:06:22.125738 3036 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.46.254:36713)
I20260925 10:06:22.134259 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.126127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.19.46.254:36713)
0925 10:06:22.126170 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.126172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.126181 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:06:22.126272 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.126273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.126284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.126345 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:22.126954 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.127467 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.128241 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.128468 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.128559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.130338 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:22.130341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:22.130342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:22.130359 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:22.132776 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.133275 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.133277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.133277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.133347 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:22.133600 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:22.133601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:22.133601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:22.133744 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:22.133880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.134034 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.134138 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:06:22.135838 3111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:06:22.137537 3111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:06:22.140313 3233 tablet_service.cc:1467] Processing CreateTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd (DEFAULT_TABLE table=test-table [id=4f70bddf279f4bd382b0aa49ff6334ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:22.140661 3233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f214cc87cde4b4ea8b7c159413828cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:22.142436 3191 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap starting.
I20260925 10:06:22.142825 3191 tablet_bootstrap.cc:654] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:22.143553 3191 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: No bootstrap required, opened a new log
I20260925 10:06:22.143636 3191 ts_tablet_manager.cc:1397] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:22.144016 3191 raft_consensus.cc:348] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.144119 3191 raft_consensus.cc:374] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:22.144191 3191 raft_consensus.cc:729] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Initialized, Role: FOLLOWER
I20260925 10:06:22.144256 3191 consensus_queue.cc:260] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.144364 3191 ts_tablet_manager.cc:1428] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:22.145723 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.140385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.19.46.194:34949)
0925 10:06:22.140671 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.140679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.141100 (+ 421us) server_negotiation.cc:407] Connection header received
0925 10:06:22.141172 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.141176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.141198 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.141420 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:22.141687 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.142684 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.144493 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.145328 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.145393 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.145493 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.145615 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.145654 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":180,"threads_started":1}
I20260925 10:06:22.145751 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.140159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34949 (local address 127.0.0.1:46820)
0925 10:06:22.140884 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.140924 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:22.141005 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:22.141533 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:22.141534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:22.141538 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:22.141597 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.141599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.142827 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.142828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.144409 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.144413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.145407 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.145409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.145504 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.145529 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:06:22.145600 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:06:22.145654 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":484,"thread_start_us":179,"threads_started":1}
I20260925 10:06:22.146672 3371 tablet_service.cc:1467] Processing CreateTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd (DEFAULT_TABLE table=test-table [id=4f70bddf279f4bd382b0aa49ff6334ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:22.147113 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f214cc87cde4b4ea8b7c159413828cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:22.147413 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.140159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:43475 (local address 127.0.0.1:45302)
0925 10:06:22.140242 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.140258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:22.140312 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:22.142957 (+ 2645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:22.142959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:22.142961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:22.143063 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.143065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.144659 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.144660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.145721 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.145723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.147091 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.147092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.147175 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.147179 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:06:22.147251 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:06:22.147307 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":33}
I20260925 10:06:22.147431 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.142531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.19.46.195:43475)
0925 10:06:22.142730 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.142736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.142750 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:06:22.142792 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.142795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.142814 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.142994 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:22.143132 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.144555 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.145832 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.146869 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.147051 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.147151 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.147271 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.147338 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":122,"threads_started":1}
I20260925 10:06:22.150516 3503 tablet_service.cc:1467] Processing CreateTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd (DEFAULT_TABLE table=test-table [id=4f70bddf279f4bd382b0aa49ff6334ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:22.150837 3579 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Bootstrap starting.
I20260925 10:06:22.150915 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f214cc87cde4b4ea8b7c159413828cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:22.151556 3579 tablet_bootstrap.cc:654] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:22.152195 3579 log.cc:824] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:22.153163 3579 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: No bootstrap required, opened a new log
I20260925 10:06:22.153244 3579 ts_tablet_manager.cc:1397] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:06:22.154838 3581 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Bootstrap starting.
I20260925 10:06:22.155602 3581 tablet_bootstrap.cc:654] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:22.155562 3579 raft_consensus.cc:348] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.155763 3579 raft_consensus.cc:374] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:22.155814 3579 raft_consensus.cc:729] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3fa6790a574f13b4631984034d3201, State: Initialized, Role: FOLLOWER
I20260925 10:06:22.155968 3579 consensus_queue.cc:260] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.156241 3581 log.cc:824] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:22.156316 3579 ts_tablet_manager.cc:1428] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:06:22.156350 3437 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
I20260925 10:06:22.157434 3581 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: No bootstrap required, opened a new log
I20260925 10:06:22.157616 3581 ts_tablet_manager.cc:1397] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:06:22.159404 3581 raft_consensus.cc:348] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.159578 3581 raft_consensus.cc:374] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:22.159606 3581 raft_consensus.cc:729] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470b6e548a434c94b5d6bb1d87cc0e03, State: Initialized, Role: FOLLOWER
I20260925 10:06:22.159746 3581 consensus_queue.cc:260] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.160017 3581 ts_tablet_manager.cc:1428] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:06:22.160085 3569 heartbeater.cc:503] Master 127.19.46.254:36713 was elected leader, sending a full tablet report...
I20260925 10:06:22.175887 3305 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:22.176036 3305 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.176417 3305 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
I20260925 10:06:22.180610 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.176808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:60449 (local address 127.19.46.195:43475)
0925 10:06:22.176865 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.176867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.176880 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:06:22.176912 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.176913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.176923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.177196 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:22.177372 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.178034 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.179437 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.180110 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.180202 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.180214 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.180473 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.180529 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:06:22.180617 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.176730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:49541 (local address 127.19.46.194:34949)
0925 10:06:22.176814 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.176816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.176997 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:06:22.177180 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.177182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.177195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.177266 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:22.177741 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.178324 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.179482 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.180157 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.180250 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.180260 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.180473 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.180529 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:06:22.180624 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.176643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:43475 (local address 127.19.46.193:60449)
0925 10:06:22.176730 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.176745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:22.176810 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:22.177202 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:22.177204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:22.177206 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:22.177293 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.177295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.178208 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.178209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.179328 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.179332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.180222 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.180224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.180321 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.180357 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:06:22.180436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:06:22.180535 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":11}
I20260925 10:06:22.180645 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.176648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34949 (local address 127.19.46.193:49541)
0925 10:06:22.176923 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.176932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:22.177014 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:22.177285 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:22.177286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:22.177288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:22.177382 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.177383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.178442 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.178443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.179333 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:22.179336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.180264 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.180265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:22.180337 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.180357 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:06:22.180436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:06:22.180508 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":97,"thread_start_us":121,"threads_started":1}
I20260925 10:06:22.181037 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f214cc87cde4b4ea8b7c159413828cd" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" is_pre_election: true
I20260925 10:06:22.181232 3523 raft_consensus.cc:2454] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 0.
I20260925 10:06:22.181439 3391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f214cc87cde4b4ea8b7c159413828cd" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3fa6790a574f13b4631984034d3201" is_pre_election: true
I20260925 10:06:22.181640 3187 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 470b6e548a434c94b5d6bb1d87cc0e03, 7449d8cdfef14036977b1c3e672e50a8; no voters:
I20260925 10:06:22.181658 3391 raft_consensus.cc:2454] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 0.
I20260925 10:06:22.181818 3305 raft_consensus.cc:2781] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:06:22.181890 3305 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:06:22.181934 3305 raft_consensus.cc:3037] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:22.182744 3305 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.182869 3305 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
I20260925 10:06:22.183101 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f214cc87cde4b4ea8b7c159413828cd" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
I20260925 10:06:22.183101 3391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f214cc87cde4b4ea8b7c159413828cd" candidate_uuid: "7449d8cdfef14036977b1c3e672e50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3fa6790a574f13b4631984034d3201"
I20260925 10:06:22.183194 3391 raft_consensus.cc:3037] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:22.183194 3523 raft_consensus.cc:3037] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:22.184156 3523 raft_consensus.cc:2454] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 1.
I20260925 10:06:22.184156 3391 raft_consensus.cc:2454] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7449d8cdfef14036977b1c3e672e50a8 in term 1.
I20260925 10:06:22.184402 3188 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3fa6790a574f13b4631984034d3201, 7449d8cdfef14036977b1c3e672e50a8; no voters:
I20260925 10:06:22.184696 3305 raft_consensus.cc:2781] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:06:22.184906 3305 raft_consensus.cc:686] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Running, Role: LEADER
I20260925 10:06:22.185006 3305 consensus_queue.cc:237] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:22.185979 3111 catalog_manager.cc:5495] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "7449d8cdfef14036977b1c3e672e50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } health_report { overall_health: HEALTHY } } }
I20260925 10:06:22.194626 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.191542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.19.46.193:42549)
0925 10:06:22.191612 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:22.191614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:22.191624 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:06:22.191675 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:22.191676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:22.191702 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:22.191807 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:06:22.191947 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.192604 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.193563 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:22.193830 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:22.193897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:22.194098 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:06:22.194222 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:06:22.194307 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:22.194388 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 10:06:22.194412 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:06:22.196946 3391 raft_consensus.cc:1261] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Refusing update from remote peer 7449d8cdfef14036977b1c3e672e50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:06:22.197072 3523 raft_consensus.cc:1261] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Refusing update from remote peer 7449d8cdfef14036977b1c3e672e50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:06:22.197472 3305 consensus_queue.cc:1035] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:06:22.197640 3586 consensus_queue.cc:1035] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:06:22.200474 3588 mvcc.cc:205] Tried to move back new op lower bound from 7333194883875520512 to 7333194883830743040. Current Snapshot: MvccSnapshot[applied={T|T < 7333194883875520512}]
I20260925 10:06:22.203190 3111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:06:22.203307 3111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:06:22.204228 3111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Sending DeleteTablet for 3 replicas of tablet 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:22.204584 3371 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:22 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:46820
I20260925 10:06:22.204622 3233 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:22 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:49090
I20260925 10:06:22.204598 3503 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f214cc87cde4b4ea8b7c159413828cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:06:22 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:45302
I20260925 10:06:22.204730 3308 tablet_replica.cc:330] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: stopping tablet replica
I20260925 10:06:22.204774 3308 raft_consensus.cc:2229] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:06:22.204866 3308 raft_consensus.cc:2258] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:22.205119 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3149
I20260925 10:06:22.205253 3308 ts_tablet_manager.cc:1905] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:22.206568 3598 tablet_replica.cc:330] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: stopping tablet replica
I20260925 10:06:22.206717 3598 raft_consensus.cc:2229] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:06:22.206854 3598 raft_consensus.cc:2258] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:22.207075 3599 tablet_replica.cc:330] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: stopping tablet replica
I20260925 10:06:22.207221 3599 raft_consensus.cc:2229] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:06:22.207353 3599 raft_consensus.cc:2258] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:06:22.207854 3599 ts_tablet_manager.cc:1905] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:22.207871 3598 ts_tablet_manager.cc:1905] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:06:22.210302 3598 ts_tablet_manager.cc:1918] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:06:22.210297 3599 ts_tablet_manager.cc:1918] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:06:22.210410 3599 log.cc:1196] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:22.210412 3598 log.cc:1196] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:22.210712 3598 ts_tablet_manager.cc:1939] T 2f214cc87cde4b4ea8b7c159413828cd P 0c3fa6790a574f13b4631984034d3201: Deleting consensus metadata
I20260925 10:06:22.210870 3599 ts_tablet_manager.cc:1939] T 2f214cc87cde4b4ea8b7c159413828cd P 470b6e548a434c94b5d6bb1d87cc0e03: Deleting consensus metadata
I20260925 10:06:22.211228 3098 catalog_manager.cc:4841] TS 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): tablet 2f214cc87cde4b4ea8b7c159413828cd (table test-table [id=4f70bddf279f4bd382b0aa49ff6334ce]) successfully deleted
I20260925 10:06:22.211367 3097 catalog_manager.cc:4841] TS 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): tablet 2f214cc87cde4b4ea8b7c159413828cd (table test-table [id=4f70bddf279f4bd382b0aa49ff6334ce]) successfully deleted
W20260925 10:06:22.212468 3096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:42549 (error 108)
I20260925 10:06:22.213033 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:22.212881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:42549 (local address 127.0.0.1:49108)
0925 10:06:22.212949 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:22.212990 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:06:22.213328 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3304
W20260925 10:06:22.213222 3096 catalog_manager.cc:4584] TS 7449d8cdfef14036977b1c3e672e50a8 (127.19.46.193:42549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f214cc87cde4b4ea8b7c159413828cd: Network error: Client connection negotiation failed: client connection to 127.19.46.193:42549: connect: Connection refused (error 111)
I20260925 10:06:22.219472 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3440
I20260925 10:06:22.224565 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3078
I20260925 10:06:42.230824 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--webserver_interface=127.19.46.254
--webserver_port=41579
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:42.326395 3601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:42.326670 3601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:42.326699 3601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:42.328104 3601 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:06:42.328171 3601 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:06:42.328186 3601 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:06:42.328197 3601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:06:42.328212 3601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:42.328225 3601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:42.328238 3601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:06:42.328250 3601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:06:42.329860 3601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:42.330101 3601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:42.330309 3601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:42.337344 3609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.337400 3601 server_base.cc:1034] running on GCE node
W20260925 10:06:42.337347 3606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.337577 3607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.338434 3601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:42.339082 3601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:42.340380 3601 hybrid_clock.cc:648] HybridClock initialized: now 1790330802340341 us; error 81 us; skew 500 ppm
Sep 25 10:06:42 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:42.344995 3601 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:06:42.345810 3601 webserver.cc:460] Webserver started at http://127.19.46.254:41579/ using document root <none> and password file <none>
I20260925 10:06:42.346164 3601 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:42.346258 3601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:42.348237 3601 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:06:42.349263 3616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.349633 3601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:42.349771 3601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "1ff1653703a440c5a627014efaab98ce"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "973e006ad7e0e3d2eaf31fed5ee5ba5e"
server_key_iv: "d4238dca758d4937235cb1a6f12fbafd"
server_key_version: "encryptionkey@0"
I20260925 10:06:42.350229 3601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:42.358951 3601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:42.362741 3601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:42.362885 3601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:42.367435 3601 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:36713
I20260925 10:06:42.367573 3668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:36713 every 8 connection(s)
I20260925 10:06:42.367931 3601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:06:42.369587 3669 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:06:42.370473 3669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap starting.
I20260925 10:06:42.377678 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3601
I20260925 10:06:42.378014 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:42549
--local_ip_for_outbound_sockets=127.19.46.193
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=39575
--webserver_interface=127.19.46.193
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:06:42.380266 3669 log.cc:824] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:42.384178 3669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:06:42.384639 3669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Bootstrap complete.
I20260925 10:06:42.389289 3669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:42.389818 3669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Initialized, Role: FOLLOWER
I20260925 10:06:42.390033 3669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:42.390141 3669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:06:42.390213 3669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:06:42.390281 3669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:06:42.392136 3669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:42.392266 3669 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff1653703a440c5a627014efaab98ce; no voters:
I20260925 10:06:42.392606 3669 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:06:42.392707 3673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:06:42.392894 3673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [term 3 LEADER]: Becoming Leader. State: Replica: 1ff1653703a440c5a627014efaab98ce, State: Running, Role: LEADER
I20260925 10:06:42.392912 3669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:06:42.393056 3673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } }
I20260925 10:06:42.393354 3674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:42.393492 3674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:42.393790 3675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff1653703a440c5a627014efaab98ce. Latest consensus state: current_term: 3 leader_uuid: "1ff1653703a440c5a627014efaab98ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff1653703a440c5a627014efaab98ce" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 36713 } } }
I20260925 10:06:42.393937 3675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:42.393932 3681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:06:42.395044 3681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:06:42.395155 3681 catalog_manager.cc:1259] Loaded cluster ID: a06e8fccc0da471c8eeaf6d4039363ed
I20260925 10:06:42.395202 3681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:06:42.396227 3681 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:06:42.396445 3681 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Loaded TSK: 0
I20260925 10:06:42.396512 3681 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ff1653703a440c5a627014efaab98ce: Loaded TSK: 1
F20260925 10:06:42.396771 3681 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f552b0569ad google::LogMessage::Fail() at ??:0
@ 0x7f552b05988a google::LogMessage::SendToLog() at ??:0
@ 0x7f552b0564de google::LogMessage::Flush() at ??:0
@ 0x7f552b057a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f552d7c3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f552d7c0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5532fb01cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5532fb048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5532fb0594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5532f76c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5532f80ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f552cf240d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f552cf19954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f552dd7d6db start_thread at ??:0
@ 0x7f5529b5461f clone at ??:0
W20260925 10:06:42.507694 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:42.507972 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:42.508004 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:42.510074 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:42.510146 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:42.510185 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:06:42.512204 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=39575
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:42.512521 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:42.512801 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:42.519003 3712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.519192 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.519565 3671 server_base.cc:1034] running on GCE node
W20260925 10:06:42.519189 3710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.520251 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:42.520545 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:42.521804 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790330802521777 us; error 35 us; skew 500 ppm
Sep 25 10:06:42 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:42.525255 3671 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:06:42.525947 3671 webserver.cc:460] Webserver started at http://127.19.46.193:39575/ using document root <none> and password file <none>
I20260925 10:06:42.526211 3671 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:42.526285 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:42.528539 3671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:06:42.529672 3721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.530019 3671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:06:42.530151 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "7449d8cdfef14036977b1c3e672e50a8"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "14c263dbbc72ce563c77a32b1d987311"
server_key_iv: "d097b8dcb400818e401dbd8ec542a46f"
server_key_version: "encryptionkey@0"
I20260925 10:06:42.530555 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:42.564982 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:42.567598 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:42.567724 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:42.567941 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:42.568375 3728 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:06:42.570451 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:06:42.570549 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:06:42.570600 3671 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:06:42.571312 3671 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:06:42.571373 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:06:42.571452 3728 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap starting.
I20260925 10:06:42.573551 3728 log.cc:824] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:42.580756 3728 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:06:42.580814 3671 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:42549
I20260925 10:06:42.580987 3836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:42549 every 8 connection(s)
I20260925 10:06:42.581158 3728 tablet_bootstrap.cc:492] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Bootstrap complete.
I20260925 10:06:42.581295 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:06:42.581337 3728 ts_tablet_manager.cc:1397] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 10:06:42.583535 3728 raft_consensus.cc:348] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:42.583979 3728 raft_consensus.cc:729] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7449d8cdfef14036977b1c3e672e50a8, State: Initialized, Role: FOLLOWER
I20260925 10:06:42.584185 3728 consensus_queue.cc:260] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:42.584540 3728 ts_tablet_manager.cc:1428] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.004s
I20260925 10:06:42.586309 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3671
I20260925 10:06:42.586660 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:34949
--local_ip_for_outbound_sockets=127.19.46.194
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=44363
--webserver_interface=127.19.46.194
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:42.626565 3839 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:06:42.582382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:44869)
0925 10:06:42.582573 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.582793 (+ 220us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:42.583394 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:42.626323 (+ 42929us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
W20260925 10:06:42.627046 3837 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:36713 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:36713: Client connection negotiation failed: client connection to 127.19.46.254:36713: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:06:42.627517 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.627375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:51977)
0925 10:06:42.627440 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.627489 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:06:42.627915 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.627811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:38785)
0925 10:06:42.627864 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.627894 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:06:42.628124 3837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:06:42.690158 3841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:42.690424 3841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:42.690457 3841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:42.692215 3841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:42.692294 3841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:42.692366 3841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:06:42.694518 3841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:34949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=44363
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:42.694790 3841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:42.695029 3841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:42.701929 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.701939 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.702006 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.702371 3841 server_base.cc:1034] running on GCE node
I20260925 10:06:42.702948 3841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:42.703315 3841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:42.704525 3841 hybrid_clock.cc:648] HybridClock initialized: now 1790330802704467 us; error 76 us; skew 500 ppm
Sep 25 10:06:42 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:42.707469 3841 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:06:42.707942 3841 webserver.cc:460] Webserver started at http://127.19.46.194:44363/ using document root <none> and password file <none>
I20260925 10:06:42.708204 3841 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:42.708280 3841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:42.709956 3841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:42.711182 3856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.711623 3841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:42.711747 3841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "0c3fa6790a574f13b4631984034d3201"
format_stamp: "Formatted at 2026-09-25 10:06:00 on dist-test-slave-f9rl"
server_key: "209a1c00a85d4bab24d56bd27bdf1df4"
server_key_iv: "ca79c7210b711e9abb579c65c7b99122"
server_key_version: "encryptionkey@0"
I20260925 10:06:42.712188 3841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:42.719874 3841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:42.722404 3841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:42.722589 3841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:42.722843 3841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:42.723160 3841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:42.723214 3841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.723258 3841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:42.723284 3841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.729547 3841 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34949
I20260925 10:06:42.729615 3969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34949 every 8 connection(s)
I20260925 10:06:42.730005 3841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:06:42.731528 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.730937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:50689)
0925 10:06:42.731102 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.731178 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260925 10:06:42.731825 3970 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:36713 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:36713: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
I20260925 10:06:42.732148 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.732060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:42587)
0925 10:06:42.732104 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.732131 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:06:42.732410 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.732362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:45737)
0925 10:06:42.732382 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.732399 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 10:06:42.732525 3970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:06:42.734163 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3841
I20260925 10:06:42.734575 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:43475
--local_ip_for_outbound_sockets=127.19.46.195
--tserver_master_addrs=127.19.46.254:36713
--webserver_port=46401
--webserver_interface=127.19.46.195
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:42.830102 3973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:42.830371 3973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:42.830401 3973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:42.831786 3973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:42.831874 3973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:42.831929 3973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:06:42.833801 3973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:43475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=46401
--tserver_master_addrs=127.19.46.254:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:42.834134 3973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:42.834396 3973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:42.841089 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.841410 3979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:42.841085 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:42.841581 3973 server_base.cc:1034] running on GCE node
I20260925 10:06:42.842084 3973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:42.842422 3973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:42.843677 3973 hybrid_clock.cc:648] HybridClock initialized: now 1790330802843637 us; error 43 us; skew 500 ppm
Sep 25 10:06:42 dist-test-slave-f9rl krb5kdc[2528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:42.848065 3973 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:06:42.848927 3973 webserver.cc:460] Webserver started at http://127.19.46.195:46401/ using document root <none> and password file <none>
I20260925 10:06:42.849421 3973 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:42.849660 3973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:42.851593 3973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:06:42.852448 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.852784 3973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:06:42.852922 3973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "470b6e548a434c94b5d6bb1d87cc0e03"
format_stamp: "Formatted at 2026-09-25 10:06:01 on dist-test-slave-f9rl"
server_key: "e2435a03ef74cf74204669b4912b0cd0"
server_key_iv: "93732961c7f0a3237f69c74929c959c9"
server_key_version: "encryptionkey@0"
I20260925 10:06:42.853369 3973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:42.891912 3973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:42.895148 3973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:42.895383 3973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:42.895704 3973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:42.896100 3973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:42.896168 3973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.896219 3973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:42.896251 3973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:42.904524 3973 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:43475
I20260925 10:06:42.904574 4101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:43475 every 8 connection(s)
I20260925 10:06:42.905016 3973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:06:42.906443 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.905921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:56079)
0925 10:06:42.906213 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.906294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
W20260925 10:06:42.907248 4102 heartbeater.cc:644] Failed to heartbeat to 127.19.46.254:36713 (0 consecutive failures): Network error: Failed to ping master at 127.19.46.254:36713: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
I20260925 10:06:42.907876 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.907705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:35217)
0925 10:06:42.907804 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.907849 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 10:06:42.908370 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:42.908228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:54127)
0925 10:06:42.908304 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:42.908347 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:06:42.908655 4102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:06:42.911840 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 3973
I20260925 10:06:43.628940 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.628630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:40637)
0925 10:06:43.628839 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:43.628884 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260925 10:06:43.733600 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.733197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:57783)
0925 10:06:43.733408 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:43.733490 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260925 10:06:43.847162 4108 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:43.847307 4108 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:43.847811 4108 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
Sep 25 10:06:43 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.194@KRBTEST.COM
Sep 25 10:06:43 dist-test-slave-f9rl krb5kdc[2528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330802, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.195@KRBTEST.COM
I20260925 10:06:43.864709 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.848067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34949 (local address 127.19.46.193:38023)
0925 10:06:43.848200 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:43.848212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:43.848262 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:43.849179 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:43.849181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:43.849629 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:43.849886 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:43.849893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.851243 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.851244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:43.852413 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:43.852417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.853405 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.853406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:43.853725 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:43.854786 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:43.854804 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:43.857408 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:43.860951 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:43.860966 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:43.860969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:43.861605 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:43.862428 (+ 823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:43.862431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:43.862431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:43.862685 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:43.863797 (+ 1112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:43.863810 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:43.864003 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 10:06:43.864142 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:06:43.864445 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":32,"thread_start_us":55,"threads_started":1}
I20260925 10:06:43.864827 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.848063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:43475 (local address 127.19.46.193:56379)
0925 10:06:43.848129 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:43.848149 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:43.848388 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:43.849142 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:43.849145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:43.849629 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:43.850104 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:43.850106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.852193 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.852194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:43.853221 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:43.853225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.854882 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.854884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:43.854998 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:43.855644 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:43.855650 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:43.857830 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:43.861643 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:43.861645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:43.861645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:43.862000 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:43.862914 (+ 914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:43.862915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:43.862916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:43.863008 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:43.864304 (+ 1296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:43.864306 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:43.864383 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 10:06:43.864481 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:06:43.864610 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260925 10:06:43.864879 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.848268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:56379 (local address 127.19.46.195:43475)
0925 10:06:43.848395 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:43.848404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:43.848423 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:06:43.848870 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:43.848872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:43.848933 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:43.849065 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:43.850375 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.852002 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.853388 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.854584 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.854863 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:43.858242 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:43.858248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:43.858248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:43.858265 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:43.861506 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:43.862416 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:43.862422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:43.862423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:43.862634 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:43.863205 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:43.863207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:43.863207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:43.864012 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:43.864150 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:43.864602 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 10:06:43.864775 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":48,"threads_started":1}
I20260925 10:06:43.865228 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.848257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38023 (local address 127.19.46.194:34949)
0925 10:06:43.848467 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:43.848475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:43.848502 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:06:43.848949 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:43.848951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:43.848988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:43.849164 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:43.850046 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.851109 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.852540 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:43.853272 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:43.853401 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:43.857659 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:43.857666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:43.857666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:43.857706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:43.860812 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:43.862008 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:43.862013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:43.862013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:43.862169 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:43.862933 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:43.862934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:43.862935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:43.863299 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:43.863407 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:43.864363 (+ 956us) server_negotiation.cc:299] Negotiation successful
0925 10:06:43.864664 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
W20260925 10:06:43.866246 3725 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
W20260925 10:06:43.866451 3724 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:43.866554 3724 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7449d8cdfef14036977b1c3e672e50a8; no voters: 0c3fa6790a574f13b4631984034d3201, 470b6e548a434c94b5d6bb1d87cc0e03
I20260925 10:06:43.866807 4108 raft_consensus.cc:2726] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:06:43.909663 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:43.909199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:36281)
0925 10:06:43.909378 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:43.909590 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":114,"threads_started":1}
I20260925 10:06:44.630124 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:44.629829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:35465)
0925 10:06:44.630019 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:44.630071 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260925 10:06:44.734553 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:44.734281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:57423)
0925 10:06:44.734461 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:44.734504 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":62,"threads_started":1}
I20260925 10:06:44.910611 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:44.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:51509)
0925 10:06:44.910514 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:44.910560 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260925 10:06:45.631085 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:45.630803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:36645)
0925 10:06:45.630979 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:45.631032 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260925 10:06:45.735641 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:45.735260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:40825)
0925 10:06:45.735491 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:45.735568 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":66,"threads_started":1}
I20260925 10:06:45.761109 4118 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:45.761224 4118 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:45.761417 4118 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
W20260925 10:06:45.762063 3724 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
W20260925 10:06:45.762142 3725 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:45.762205 3725 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7449d8cdfef14036977b1c3e672e50a8; no voters: 0c3fa6790a574f13b4631984034d3201, 470b6e548a434c94b5d6bb1d87cc0e03
I20260925 10:06:45.762319 4118 raft_consensus.cc:2726] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:06:45.911648 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:45.911297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:47701)
0925 10:06:45.911551 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:45.911598 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260925 10:06:46.631987 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:46.631740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:54567)
0925 10:06:46.631887 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:46.631939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260925 10:06:46.736814 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:46.736535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:55929)
0925 10:06:46.736706 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:46.736757 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260925 10:06:46.912533 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:46.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:60997)
0925 10:06:46.912430 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:46.912480 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260925 10:06:47.632923 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:47.632638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:40853)
0925 10:06:47.632831 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:47.632872 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260925 10:06:47.737810 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:47.737544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:53397)
0925 10:06:47.737707 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:47.737758 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260925 10:06:47.913340 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:47.913079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:52649)
0925 10:06:47.913249 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:47.913292 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260925 10:06:48.037590 4126 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:48.037694 4126 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:48.037878 4126 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
W20260925 10:06:48.038669 3724 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
W20260925 10:06:48.038750 3725 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:48.038784 3724 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7449d8cdfef14036977b1c3e672e50a8; no voters: 0c3fa6790a574f13b4631984034d3201, 470b6e548a434c94b5d6bb1d87cc0e03
I20260925 10:06:48.038888 4126 raft_consensus.cc:2726] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:06:48.634289 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:48.633892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:38341)
0925 10:06:48.634159 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:48.634233 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":180,"threads_started":1}
I20260925 10:06:48.738694 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:48.738435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:50857)
0925 10:06:48.738593 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:48.738643 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260925 10:06:48.914367 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:48.914142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:51249)
0925 10:06:48.914272 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:48.914319 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260925 10:06:49.635449 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:49.635009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:40327)
0925 10:06:49.635307 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:49.635352 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260925 10:06:49.739661 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:49.739288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:32803)
0925 10:06:49.739447 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:49.739492 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260925 10:06:49.915518 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:49.915166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:37415)
0925 10:06:49.915303 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:49.915347 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 10:06:50.245661 4133 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:50.245771 4133 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:50.245957 4133 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
W20260925 10:06:50.246450 3725 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
W20260925 10:06:50.246532 3724 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:50.246568 3724 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7449d8cdfef14036977b1c3e672e50a8; no voters: 0c3fa6790a574f13b4631984034d3201, 470b6e548a434c94b5d6bb1d87cc0e03
I20260925 10:06:50.246654 4133 raft_consensus.cc:2726] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:06:50.636281 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:50.636019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:42195)
0925 10:06:50.636183 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:50.636231 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260925 10:06:50.740903 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:50.740525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:50209)
0925 10:06:50.740808 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:50.740851 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":75,"threads_started":1}
I20260925 10:06:50.916671 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:50.916190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:42279)
0925 10:06:50.916537 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:50.916595 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":68,"threads_started":1}
I20260925 10:06:51.637878 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:51.637404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:33499)
0925 10:06:51.637760 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:51.637827 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":242,"threads_started":1}
I20260925 10:06:51.741812 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:51.741555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:52189)
0925 10:06:51.741711 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:51.741759 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260925 10:06:51.917843 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:51.917577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:39501)
0925 10:06:51.917742 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:51.917791 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260925 10:06:52.638856 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:52.638607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:57233)
0925 10:06:52.638769 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:52.638808 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260925 10:06:52.742728 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:52.742426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:56303)
0925 10:06:52.742630 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:52.742675 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260925 10:06:52.919061 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:52.918737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:45301)
0925 10:06:52.918960 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:52.919009 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260925 10:06:53.639894 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:53.639595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:34223)
0925 10:06:53.639799 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:53.639841 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":137,"threads_started":1}
I20260925 10:06:53.743647 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:53.743309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:57445)
0925 10:06:53.743540 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:53.743586 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260925 10:06:53.920559 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:53.919861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:39311)
0925 10:06:53.920181 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:53.920490 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260925 10:06:54.641003 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:54.640751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:56663)
0925 10:06:54.640904 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:54.640953 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260925 10:06:54.744536 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:54.744281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:54335)
0925 10:06:54.744433 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:54.744485 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260925 10:06:54.921891 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:54.921420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:57243)
0925 10:06:54.921785 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:54.921832 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260925 10:06:55.642524 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:55.642112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:59073)
0925 10:06:55.642364 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:55.642447 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260925 10:06:55.745568 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:55.745206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:48153)
0925 10:06:55.745389 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:55.745466 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260925 10:06:55.923002 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:55.922729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:52409)
0925 10:06:55.922904 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:55.922951 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260925 10:06:56.643734 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:56.643420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:44377)
0925 10:06:56.643625 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:56.643681 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260925 10:06:56.746500 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:56.746218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:37351)
0925 10:06:56.746397 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:56.746449 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260925 10:06:56.924209 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:56.923898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:43513)
0925 10:06:56.924109 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:56.924158 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260925 10:06:57.548697 4155 raft_consensus.cc:482] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:06:57.548808 4155 raft_consensus.cc:504] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3fa6790a574f13b4631984034d3201" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34949 } } peers { permanent_uuid: "470b6e548a434c94b5d6bb1d87cc0e03" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 43475 } } peers { permanent_uuid: "7449d8cdfef14036977b1c3e672e50a8" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 42549 } }
I20260925 10:06:57.549007 4155 leader_election.cc:288] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949), 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475)
W20260925 10:06:57.549938 3724 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 470b6e548a434c94b5d6bb1d87cc0e03 (127.19.46.195:43475): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
W20260925 10:06:57.550114 3725 leader_election.cc:341] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c3fa6790a574f13b4631984034d3201 (127.19.46.194:34949): Not found: Tablet not found: 2f214cc87cde4b4ea8b7c159413828cd
I20260925 10:06:57.550154 3725 leader_election.cc:302] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7449d8cdfef14036977b1c3e672e50a8; no voters: 0c3fa6790a574f13b4631984034d3201, 470b6e548a434c94b5d6bb1d87cc0e03
I20260925 10:06:57.550237 4155 raft_consensus.cc:2726] T 2f214cc87cde4b4ea8b7c159413828cd P 7449d8cdfef14036977b1c3e672e50a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:06:57.644801 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:57.644521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.193:49483)
0925 10:06:57.644686 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:57.644736 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260925 10:06:57.747534 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:57.747277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.194:45187)
0925 10:06:57.747418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:57.747455 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260925 10:06:57.920099 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3671
I20260925 10:06:57.925302 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:57.924988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:36713 (local address 127.19.46.195:33235)
0925 10:06:57.925186 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:57.925229 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.254:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260925 10:06:57.925547 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3841
I20260925 10:06:57.930629 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 3973
2026-09-25T10:06:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59017 ms)
[----------] 30 tests from SecurityITest (261655 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:06:58 dist-test-slave-f9rl krb5kdc[4161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:06:58 dist-test-slave-f9rl krb5kdc[4161](info): set up 2 sockets
Sep 25 10:06:58 dist-test-slave-f9rl krb5kdc[4161](info): commencing operation
krb5kdc: starting...
W20260925 10:06:59.047358 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:06:59.189891 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38795
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:59.289337 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:59.289674 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:59.289702 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:59.291079 4174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:06:59.291141 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:59.291157 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:59.291170 4174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:06:59.291182 4174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:06:59.292887 4174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:59.293203 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:59.293534 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:59.300635 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.301097 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.301167 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.301252 4174 server_base.cc:1034] running on GCE node
I20260925 10:06:59.301599 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:59.302023 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:59.303225 4174 hybrid_clock.cc:648] HybridClock initialized: now 1790330819303175 us; error 74 us; skew 500 ppm
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:59.306483 4174 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:06:59.307672 4174 webserver.cc:460] Webserver started at http://127.19.46.254:39327/ using document root <none> and password file <none>
I20260925 10:06:59.307947 4174 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:59.308015 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:59.308149 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:59.310015 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4c206eb876c348d798217083441876db"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "a18307af97d986df95204b9801f73b90"
server_key_iv: "e11cc9202d7b396609fbe3f6e0067cab"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.310570 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4c206eb876c348d798217083441876db"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "a18307af97d986df95204b9801f73b90"
server_key_iv: "e11cc9202d7b396609fbe3f6e0067cab"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.313052 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:06:59.314327 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.314670 4174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:59.314810 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4c206eb876c348d798217083441876db"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "a18307af97d986df95204b9801f73b90"
server_key_iv: "e11cc9202d7b396609fbe3f6e0067cab"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.314913 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:59.322247 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:59.325987 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:59.326190 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:59.330782 4174 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:38795
I20260925 10:06:59.331084 4241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:38795 every 8 connection(s)
I20260925 10:06:59.331266 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:06:59.332301 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:59.335400 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: Bootstrap starting.
I20260925 10:06:59.336334 4242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: Neither blocks nor log segments found. Creating new log.
I20260925 10:06:59.336668 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4174
I20260925 10:06:59.336791 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:06:59.336804 4242 log.cc:824] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: Log is configured to *not* fsync() on all Append() calls
I20260925 10:06:59.337049 19643 external_mini_cluster.cc:1428] Setting key 8ba92d85bdf3acf5bf0a61b22bdd11ba
I20260925 10:06:59.337911 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: No bootstrap required, opened a new log
I20260925 10:06:59.342061 4242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } }
I20260925 10:06:59.342386 4242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:06:59.342428 4242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c206eb876c348d798217083441876db, State: Initialized, Role: FOLLOWER
I20260925 10:06:59.342808 4242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } }
I20260925 10:06:59.342948 4242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:06:59.342991 4242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:06:59.343176 4242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:06:59.344025 4242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } }
I20260925 10:06:59.344265 4242 leader_election.cc:302] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c206eb876c348d798217083441876db; no voters:
I20260925 10:06:59.344493 4242 leader_election.cc:288] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:06:59.344564 4247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:06:59.344758 4247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [term 1 LEADER]: Becoming Leader. State: Replica: 4c206eb876c348d798217083441876db, State: Running, Role: LEADER
I20260925 10:06:59.344868 4242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:06:59.344956 4247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } }
I20260925 10:06:59.345371 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c206eb876c348d798217083441876db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } } }
I20260925 10:06:59.345486 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:59.345755 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c206eb876c348d798217083441876db. Latest consensus state: current_term: 1 leader_uuid: "4c206eb876c348d798217083441876db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c206eb876c348d798217083441876db" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38795 } } }
I20260925 10:06:59.345862 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db [sys.catalog]: This master's current role is: LEADER
I20260925 10:06:59.345937 4253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:06:59.346521 4253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:59.348542 4253 catalog_manager.cc:1347] Generated new cluster ID: 8af227161e284e12b8ad67c97596a660
I20260925 10:06:59.348608 4253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:06:59.353021 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.337706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.19.46.254:38795)
0925 10:06:59.337884 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.337904 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.337938 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:06:59.338487 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.338495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.338559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.339023 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:59.340672 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.341762 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.343154 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.343692 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.343785 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.347171 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:59.347176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:59.347177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:59.347190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:59.350785 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.351287 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.351291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.351291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.351400 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.351639 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.351640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.351640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.351983 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:59.352181 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.352254 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.352410 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260925 10:06:59.374998 4253 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:06:59.375635 4253 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:06:59.378432 4253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: Generated new TSK 0
I20260925 10:06:59.378832 4253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:06:59.418783 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:59.517254 4268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:59.517566 4268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:59.517607 4268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:59.519933 4268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:59.520054 4268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:59.520098 4268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:06:59.523303 4268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:59.523715 4268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:59.524037 4268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:59.529829 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.529819 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.529866 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.530205 4268 server_base.cc:1034] running on GCE node
I20260925 10:06:59.530471 4268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:59.530758 4268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:59.531973 4268 hybrid_clock.cc:648] HybridClock initialized: now 1790330819531880 us; error 116 us; skew 500 ppm
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:59.535542 4268 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:06:59.536036 4268 webserver.cc:460] Webserver started at http://127.19.46.193:45427/ using document root <none> and password file <none>
I20260925 10:06:59.536274 4268 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:59.536345 4268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:59.536532 4268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:59.537804 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "d32a97c7bc784af4b1e8ea15a45c9454"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "73996654eb403c3398b40a1fc1b82e9d"
server_key_iv: "5aa1aa73eaa14e8faa0f162144115750"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.538183 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "d32a97c7bc784af4b1e8ea15a45c9454"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "73996654eb403c3398b40a1fc1b82e9d"
server_key_iv: "5aa1aa73eaa14e8faa0f162144115750"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.539682 4268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:06:59.540685 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.540951 4268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:59.541079 4268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "d32a97c7bc784af4b1e8ea15a45c9454"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "73996654eb403c3398b40a1fc1b82e9d"
server_key_iv: "5aa1aa73eaa14e8faa0f162144115750"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.541167 4268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:59.564410 4268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:59.567215 4268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:59.567414 4268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:59.567703 4268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:59.568056 4268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:59.568113 4268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.568150 4268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:59.568198 4268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.574730 4268 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:44515
I20260925 10:06:59.574887 4396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:44515 every 8 connection(s)
I20260925 10:06:59.575251 4268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:59.585561 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4268
I20260925 10:06:59.585706 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:06:59.585938 19643 external_mini_cluster.cc:1428] Setting key 59b34c7ec16a1619b29e2035eb9204b7
I20260925 10:06:59.588930 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.576191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:55163 (local address 127.19.46.254:38795)
0925 10:06:59.576279 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.576281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.576449 (+ 168us) server_negotiation.cc:407] Connection header received
0925 10:06:59.576954 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.576955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.576967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.577108 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:59.579074 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.579667 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.580900 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.581152 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.581246 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.584466 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:59.584474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:59.584474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:59.584485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:59.586958 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.587674 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.587675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.587676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.587739 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.588106 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.588107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.588108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.588245 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:59.588330 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.588706 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.588822 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:06:59.589179 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.576137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38795 (local address 127.19.46.193:55163)
0925 10:06:59.576296 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.576338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.576836 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.577321 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.577330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.578498 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:59.578844 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.578853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.579805 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.579806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.580737 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.580741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.581257 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.581258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.581407 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.582270 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:59.582285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:59.584297 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:59.587127 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.587144 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.587148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.587483 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.587828 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.587829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.587830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.587965 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.588335 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:59.588344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:59.588539 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.588668 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.588874 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 10:06:59.589816 4397 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38795
I20260925 10:06:59.590103 4397 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:59.590560 4397 heartbeater.cc:511] Master 127.19.46.254:38795 requested a full tablet report, sending...
I20260925 10:06:59.591281 4206 ts_manager.cc:194] Registered new tserver with Master: d32a97c7bc784af4b1e8ea15a45c9454 (127.19.46.193:44515)
I20260925 10:06:59.591938 4206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:55163
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:06:59.617604 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:59.704669 4402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:59.704892 4402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:59.704919 4402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:59.706126 4402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:59.706171 4402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:59.706200 4402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:06:59.707647 4402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:59.707906 4402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:59.708130 4402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:59.714102 4408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.714089 4410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.714187 4402 server_base.cc:1034] running on GCE node
W20260925 10:06:59.714090 4407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.714702 4402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:59.715042 4402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:59.716236 4402 hybrid_clock.cc:648] HybridClock initialized: now 1790330819716218 us; error 31 us; skew 500 ppm
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:59.719491 4402 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:06:59.720078 4402 webserver.cc:460] Webserver started at http://127.19.46.194:43697/ using document root <none> and password file <none>
I20260925 10:06:59.720340 4402 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:59.720402 4402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:59.720551 4402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:59.721889 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "c8efa903bada475ca68c7c9f760bc857"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "c390e97dc6c6b78f2c2a9308120f3183"
server_key_iv: "4d0b92f83bc91e0d3aed2ced13a44bce"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.722302 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "c8efa903bada475ca68c7c9f760bc857"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "c390e97dc6c6b78f2c2a9308120f3183"
server_key_iv: "4d0b92f83bc91e0d3aed2ced13a44bce"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.724190 4402 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:06:59.725364 4417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.725968 4402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:59.726120 4402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "c8efa903bada475ca68c7c9f760bc857"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "c390e97dc6c6b78f2c2a9308120f3183"
server_key_iv: "4d0b92f83bc91e0d3aed2ced13a44bce"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.726214 4402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:59.749567 4402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:59.752740 4402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:59.752945 4402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:59.753233 4402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:59.753684 4402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:59.753778 4402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.753826 4402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:59.753857 4402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.760396 4402 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:34677
I20260925 10:06:59.760576 4530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:34677 every 8 connection(s)
I20260925 10:06:59.760910 4402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:06:59.764055 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4402
I20260925 10:06:59.764225 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:06:59.764489 19643 external_mini_cluster.cc:1428] Setting key e9bac357ecec9da50600b92238251ba9
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:06:59.774456 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.761980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:37913 (local address 127.19.46.254:38795)
0925 10:06:59.762063 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.762066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.762351 (+ 285us) server_negotiation.cc:407] Connection header received
0925 10:06:59.762823 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.762824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.762837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.762909 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:59.764816 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.765624 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.766849 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.767184 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.767263 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.770222 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:59.770223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:59.770224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:59.770231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:59.771912 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.772551 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.772552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.772552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.772846 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.773359 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.773360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.773361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.773665 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:59.773806 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.774291 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.774383 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:06:59.774519 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.761883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38795 (local address 127.19.46.194:37913)
0925 10:06:59.762130 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.762176 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.762699 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.763113 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.763119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.764147 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:59.764560 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.764570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.765788 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.765790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.766733 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.766736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.767294 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.767295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.767430 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.768065 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:59.768075 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:59.770078 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:59.772021 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.772033 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.772036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.772414 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.773001 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.773003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.773003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.773233 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.773805 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:59.773815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:59.774034 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.774177 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.774331 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":164,"threads_started":1}
I20260925 10:06:59.775784 4531 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38795
I20260925 10:06:59.776010 4531 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:59.776460 4531 heartbeater.cc:511] Master 127.19.46.254:38795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
I20260925 10:06:59.777163 4206 ts_manager.cc:194] Registered new tserver with Master: c8efa903bada475ca68c7c9f760bc857 (127.19.46.194:34677)
I20260925 10:06:59.777724 4206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:37913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:06:59.798455 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:06:59.885298 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:06:59.885540 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:06:59.885617 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:06:59.886852 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:06:59.886912 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:06:59.886942 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:06:59.888556 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:06:59.888790 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:06:59.888996 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:06:59.895993 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:06:59.896097 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.896214 4536 server_base.cc:1034] running on GCE node
W20260925 10:06:59.895993 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:06:59.896572 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:06:59.896873 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:06:59.898084 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790330819898026 us; error 82 us; skew 500 ppm
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:06:59.901633 4536 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:06:59.902168 4536 webserver.cc:460] Webserver started at http://127.19.46.195:36097/ using document root <none> and password file <none>
I20260925 10:06:59.902432 4536 fs_manager.cc:359] Metadata directory not provided
I20260925 10:06:59.902514 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:06:59.902665 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:06:59.903878 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "5f705eff9b1c438c97da499bae96bb14"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "47cfe21fed12be19dbb0f9d95cf3b850"
server_key_iv: "a06368495a241e38a2bfc2f61b096824"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.904295 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "5f705eff9b1c438c97da499bae96bb14"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "47cfe21fed12be19dbb0f9d95cf3b850"
server_key_iv: "a06368495a241e38a2bfc2f61b096824"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.907017 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 10:06:59.908420 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.908669 4536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:06:59.909030 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "5f705eff9b1c438c97da499bae96bb14"
format_stamp: "Formatted at 2026-09-25 10:06:59 on dist-test-slave-f9rl"
server_key: "47cfe21fed12be19dbb0f9d95cf3b850"
server_key_iv: "a06368495a241e38a2bfc2f61b096824"
server_key_version: "encryptionkey@0"
I20260925 10:06:59.909132 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:06:59.925863 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:06:59.931413 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:06:59.931631 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:06:59.931985 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:06:59.932487 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:06:59.932773 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.932843 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:06:59.932866 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:06:59.944304 4536 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:42101
I20260925 10:06:59.944459 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:42101 every 8 connection(s)
I20260925 10:06:59.944888 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:06:59.954828 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4536
I20260925 10:06:59.954952 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:06:59.955184 19643 external_mini_cluster.cc:1428] Setting key 6de5c835c7389433f19ad3f376d9927a
Sep 25 10:06:59 dist-test-slave-f9rl krb5kdc[4161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330819, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:06:59.961573 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.945868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:34467 (local address 127.19.46.254:38795)
0925 10:06:59.945994 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.945997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.946432 (+ 435us) server_negotiation.cc:407] Connection header received
0925 10:06:59.947596 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.947598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.947615 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.947704 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:59.949763 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.950684 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.952523 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.952924 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.953033 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.956211 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:59.956214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:59.956214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:59.956224 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:59.958572 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.959546 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.959548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.959549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.959640 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.960134 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.960135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.960136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.960334 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:59.960431 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.961296 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.961402 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:06:59.961623 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.945791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:38795 (local address 127.19.46.195:34467)
0925 10:06:59.946072 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.946282 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.947449 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.947823 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.947842 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.949033 (+ 1191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:06:59.949518 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.949526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.950841 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.950842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.952335 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.952344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.953131 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.953132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.953364 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.954114 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:06:59.954127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:06:59.956094 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:06:59.958795 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.958816 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.958821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.959407 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.959789 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:06:59.959790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:06:59.959791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:06:59.960025 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:06:59.960494 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:06:59.960512 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:06:59.960790 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.961076 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.961281 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":153,"threads_started":1}
I20260925 10:06:59.962230 4666 heartbeater.cc:347] Connected to a master server at 127.19.46.254:38795
I20260925 10:06:59.962368 4666 heartbeater.cc:464] Registering TS with master...
I20260925 10:06:59.962607 4666 heartbeater.cc:511] Master 127.19.46.254:38795 requested a full tablet report, sending...
I20260925 10:06:59.963284 4206 ts_manager.cc:194] Registered new tserver with Master: 5f705eff9b1c438c97da499bae96bb14 (127.19.46.195:42101)
I20260925 10:06:59.963843 4206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:34467
I20260925 10:06:59.967648 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:06:59.978492 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.969268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.19.46.254:38795)
0925 10:06:59.969460 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.969462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.969564 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:06:59.969718 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.969720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.969734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.969800 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:06:59.970529 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.971143 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.972009 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.972322 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.972410 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.974397 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:06:59.974400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:06:59.974401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:06:59.974414 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:06:59.976659 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.977347 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.977349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.977350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.977490 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:06:59.977809 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:06:59.977810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:06:59.977810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:06:59.978002 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:06:59.978128 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.978266 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.978370 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 10:06:59.981424 4206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:06:59.984071 4206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:06:59.994266 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.988072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44515 (local address 127.0.0.1:33276)
0925 10:06:59.988841 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.988853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.988907 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.989609 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.989610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.989614 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:59.989685 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.989687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.991246 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.991247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.992700 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.992704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.993846 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.993847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.993957 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.993984 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.994044 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.994132 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":231,"thread_start_us":106,"threads_started":1}
I20260925 10:06:59.994431 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.987757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:34677 (local address 127.0.0.1:39730)
0925 10:06:59.987954 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.988006 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.988098 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.989263 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.989264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.989267 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:59.989397 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.989401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.991270 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.991271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.993016 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.993020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.994120 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.994121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.994201 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.994207 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.994265 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.994329 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":216,"thread_start_us":129,"threads_started":1}
I20260925 10:06:59.994433 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.988099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.19.46.194:34677)
0925 10:06:59.988762 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.988771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.988789 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:06:59.988844 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.988852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.988875 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.989263 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:59.989550 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.991019 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.993146 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.994000 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.994106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.994182 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.994288 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.994331 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":133,"threads_started":1}
I20260925 10:06:59.994831 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.987783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:42101 (local address 127.0.0.1:58782)
0925 10:06:59.988619 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 10:06:59.988637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:06:59.988731 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:06:59.989231 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:06:59.989235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:06:59.989245 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:06:59.989367 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.989369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.991273 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.991274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.993016 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:06:59.993020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.994558 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.994559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:06:59.994627 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.994637 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:06:59.994720 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:06:59.994770 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":205,"thread_start_us":130,"threads_started":1}
I20260925 10:06:59.994946 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.988571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.19.46.193:44515)
0925 10:06:59.989390 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.989395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.989405 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:06:59.989480 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.989482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.989499 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.989599 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:59.989750 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.990817 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.992848 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.993703 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.994662 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.994758 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.994789 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.994850 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":119,"threads_started":1}
I20260925 10:06:59.994946 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:06:59.988346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.19.46.195:42101)
0925 10:06:59.989028 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 10:06:59.989033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:06:59.989042 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:06:59.989084 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:06:59.989085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:06:59.989099 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:06:59.989204 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:06:59.989513 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.991006 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.993184 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:06:59.993991 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:06:59.994074 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:06:59.994145 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:06:59.994759 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 10:06:59.994845 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":146,"threads_started":1}
I20260925 10:06:59.995842 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec (DEFAULT_TABLE table=test-table [id=6c22a0ea81364ff3a809e0d6dc5c6472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:59.995891 4465 tablet_service.cc:1467] Processing CreateTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec (DEFAULT_TABLE table=test-table [id=6c22a0ea81364ff3a809e0d6dc5c6472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:59.995963 4599 tablet_service.cc:1467] Processing CreateTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec (DEFAULT_TABLE table=test-table [id=6c22a0ea81364ff3a809e0d6dc5c6472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:06:59.996308 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e57653b0c44e3f8af37fbbdffb1bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:59.996313 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e57653b0c44e3f8af37fbbdffb1bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:06:59.996573 4465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e57653b0c44e3f8af37fbbdffb1bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:00.000450 4685 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Bootstrap starting.
I20260925 10:07:00.000519 4683 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Bootstrap starting.
I20260925 10:07:00.001225 4685 tablet_bootstrap.cc:654] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.001376 4683 tablet_bootstrap.cc:654] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.001927 4685 log.cc:824] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.002367 4684 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Bootstrap starting.
I20260925 10:07:00.002404 4683 log.cc:824] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.003188 4684 tablet_bootstrap.cc:654] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.003213 4685 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: No bootstrap required, opened a new log
I20260925 10:07:00.003275 4685 ts_tablet_manager.cc:1397] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:07:00.003530 4683 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: No bootstrap required, opened a new log
I20260925 10:07:00.003962 4683 ts_tablet_manager.cc:1397] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:07:00.004453 4684 log.cc:824] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.005513 4685 raft_consensus.cc:348] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.005743 4685 raft_consensus.cc:374] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.005789 4685 raft_consensus.cc:729] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32a97c7bc784af4b1e8ea15a45c9454, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.005787 4684 tablet_bootstrap.cc:492] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: No bootstrap required, opened a new log
I20260925 10:07:00.005868 4684 ts_tablet_manager.cc:1397] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:07:00.005936 4685 consensus_queue.cc:260] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.006273 4685 ts_tablet_manager.cc:1428] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 10:07:00.006453 4397 heartbeater.cc:503] Master 127.19.46.254:38795 was elected leader, sending a full tablet report...
I20260925 10:07:00.007818 4683 raft_consensus.cc:348] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.008056 4683 raft_consensus.cc:374] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.008100 4683 raft_consensus.cc:729] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8efa903bada475ca68c7c9f760bc857, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.008284 4683 consensus_queue.cc:260] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.008749 4531 heartbeater.cc:503] Master 127.19.46.254:38795 was elected leader, sending a full tablet report...
I20260925 10:07:00.008836 4683 ts_tablet_manager.cc:1428] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:07:00.010303 4684 raft_consensus.cc:348] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.010653 4684 raft_consensus.cc:374] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.010707 4684 raft_consensus.cc:729] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f705eff9b1c438c97da499bae96bb14, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.010885 4684 consensus_queue.cc:260] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.011257 4684 ts_tablet_manager.cc:1428] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:07:00.011399 4666 heartbeater.cc:503] Master 127.19.46.254:38795 was elected leader, sending a full tablet report...
W20260925 10:07:00.011754 4532 tablet.cc:2369] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:07:00.076148 4398 tablet.cc:2369] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:00.076114 4690 raft_consensus.cc:482] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:00.076275 4690 raft_consensus.cc:504] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.076723 4690 leader_election.cc:288] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f705eff9b1c438c97da499bae96bb14 (127.19.46.195:42101), d32a97c7bc784af4b1e8ea15a45c9454 (127.19.46.193:44515)
I20260925 10:07:00.079872 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.077168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:48091 (local address 127.19.46.195:42101)
0925 10:07:00.077218 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.077221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.077271 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:07:00.077338 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.077339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.077347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.077421 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.077639 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.078110 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079099 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.079504 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079603 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.079612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.079763 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.079798 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:07:00.079885 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.077004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:42101 (local address 127.19.46.194:48091)
0925 10:07:00.077215 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.077227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.077281 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.077424 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.077425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.077426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.077517 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.077519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.078195 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.078196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.079004 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.079006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.079603 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.079677 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.079683 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.079756 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.079798 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":135,"threads_started":1}
I20260925 10:07:00.079972 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.076912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44515 (local address 127.19.46.194:52047)
0925 10:07:00.076957 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.076970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.077018 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.077268 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.077270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.077275 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.077356 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.077358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.078179 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.078180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.079029 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.079032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.079661 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.079757 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.079760 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.079829 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.079888 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":6}
I20260925 10:07:00.079970 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.077020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52047 (local address 127.19.46.193:44515)
0925 10:07:00.077103 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.077106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.077118 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:00.077154 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.077155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.077169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.077249 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.077461 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.078080 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079138 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.079566 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.079662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.079672 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.079850 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.079893 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:07:00.080269 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e57653b0c44e3f8af37fbbdffb1bec" candidate_uuid: "c8efa903bada475ca68c7c9f760bc857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f705eff9b1c438c97da499bae96bb14" is_pre_election: true
I20260925 10:07:00.080456 4619 raft_consensus.cc:2454] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8efa903bada475ca68c7c9f760bc857 in term 0.
I20260925 10:07:00.080413 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e57653b0c44e3f8af37fbbdffb1bec" candidate_uuid: "c8efa903bada475ca68c7c9f760bc857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" is_pre_election: true
I20260925 10:07:00.080598 4351 raft_consensus.cc:2454] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8efa903bada475ca68c7c9f760bc857 in term 0.
I20260925 10:07:00.080694 4421 leader_election.cc:302] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f705eff9b1c438c97da499bae96bb14, c8efa903bada475ca68c7c9f760bc857; no voters:
I20260925 10:07:00.080819 4690 raft_consensus.cc:2781] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:00.080876 4690 raft_consensus.cc:482] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:00.080900 4690 raft_consensus.cc:3037] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.081691 4690 raft_consensus.cc:504] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.081844 4690 leader_election.cc:288] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [CANDIDATE]: Term 1 election: Requested vote from peers 5f705eff9b1c438c97da499bae96bb14 (127.19.46.195:42101), d32a97c7bc784af4b1e8ea15a45c9454 (127.19.46.193:44515)
I20260925 10:07:00.082019 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e57653b0c44e3f8af37fbbdffb1bec" candidate_uuid: "c8efa903bada475ca68c7c9f760bc857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f705eff9b1c438c97da499bae96bb14"
I20260925 10:07:00.082067 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e57653b0c44e3f8af37fbbdffb1bec" candidate_uuid: "c8efa903bada475ca68c7c9f760bc857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32a97c7bc784af4b1e8ea15a45c9454"
I20260925 10:07:00.082130 4619 raft_consensus.cc:3037] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.082140 4351 raft_consensus.cc:3037] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.083146 4619 raft_consensus.cc:2454] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8efa903bada475ca68c7c9f760bc857 in term 1.
I20260925 10:07:00.083146 4351 raft_consensus.cc:2454] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8efa903bada475ca68c7c9f760bc857 in term 1.
I20260925 10:07:00.083425 4419 leader_election.cc:302] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8efa903bada475ca68c7c9f760bc857, d32a97c7bc784af4b1e8ea15a45c9454; no voters:
I20260925 10:07:00.083580 4690 raft_consensus.cc:2781] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:00.083775 4690 raft_consensus.cc:686] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 1 LEADER]: Becoming Leader. State: Replica: c8efa903bada475ca68c7c9f760bc857, State: Running, Role: LEADER
I20260925 10:07:00.084149 4690 consensus_queue.cc:237] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } }
I20260925 10:07:00.085646 4205 catalog_manager.cc:5495] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8efa903bada475ca68c7c9f760bc857 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "c8efa903bada475ca68c7c9f760bc857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8efa903bada475ca68c7c9f760bc857" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 34677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 } health_report { overall_health: UNKNOWN } } }
I20260925 10:07:00.113654 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.110629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.19.46.194:34677)
0925 10:07:00.110725 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.110726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.110734 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:07:00.110770 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.110771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.110793 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.110877 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:00.111034 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.111543 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.112726 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.112933 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.113030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.113215 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:00.113333 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:00.113433 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.113507 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.113532 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:07:00.116345 4351 raft_consensus.cc:1261] T 20e57653b0c44e3f8af37fbbdffb1bec P d32a97c7bc784af4b1e8ea15a45c9454 [term 1 FOLLOWER]: Refusing update from remote peer c8efa903bada475ca68c7c9f760bc857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:00.116318 4619 raft_consensus.cc:1261] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 1 FOLLOWER]: Refusing update from remote peer c8efa903bada475ca68c7c9f760bc857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:00.116889 4693 consensus_queue.cc:1035] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f705eff9b1c438c97da499bae96bb14" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:00.117167 4690 consensus_queue.cc:1035] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32a97c7bc784af4b1e8ea15a45c9454" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:00.119527 4696 mvcc.cc:205] Tried to move back new op lower bound from 7333195039191994368 to 7333195039065903104. Current Snapshot: MvccSnapshot[applied={T|T < 7333195039191994368}]
I20260925 10:07:00.119937 4698 mvcc.cc:205] Tried to move back new op lower bound from 7333195039191994368 to 7333195039065903104. Current Snapshot: MvccSnapshot[applied={T|T < 7333195039191994368}]
I20260925 10:07:00.122524 4205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:00.122618 4205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:00.123601 4205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c206eb876c348d798217083441876db: Sending DeleteTablet for 3 replicas of tablet 20e57653b0c44e3f8af37fbbdffb1bec
I20260925 10:07:00.123917 4465 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:39730
I20260925 10:07:00.123936 4599 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:58782
I20260925 10:07:00.123997 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e57653b0c44e3f8af37fbbdffb1bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:00 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:33276
I20260925 10:07:00.124501 4707 tablet_replica.cc:330] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: stopping tablet replica
I20260925 10:07:00.124642 4707 raft_consensus.cc:2229] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:00.124650 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4268
I20260925 10:07:00.124825 4707 raft_consensus.cc:2258] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:00.125391 4707 ts_tablet_manager.cc:1905] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:00.127449 4706 tablet_replica.cc:330] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: stopping tablet replica
I20260925 10:07:00.127660 4706 raft_consensus.cc:2229] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:00.127821 4707 ts_tablet_manager.cc:1918] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:00.127926 4707 log.cc:1196] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/20e57653b0c44e3f8af37fbbdffb1bec
I20260925 10:07:00.128289 4707 ts_tablet_manager.cc:1939] T 20e57653b0c44e3f8af37fbbdffb1bec P c8efa903bada475ca68c7c9f760bc857: Deleting consensus metadata
I20260925 10:07:00.128845 4191 catalog_manager.cc:4841] TS c8efa903bada475ca68c7c9f760bc857 (127.19.46.194:34677): tablet 20e57653b0c44e3f8af37fbbdffb1bec (table test-table [id=6c22a0ea81364ff3a809e0d6dc5c6472]) successfully deleted
I20260925 10:07:00.129020 4706 raft_consensus.cc:2258] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:00.130281 4706 ts_tablet_manager.cc:1905] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:00.132812 4706 ts_tablet_manager.cc:1918] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:00.132925 4706 log.cc:1196] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/20e57653b0c44e3f8af37fbbdffb1bec
I20260925 10:07:00.133252 4706 ts_tablet_manager.cc:1939] T 20e57653b0c44e3f8af37fbbdffb1bec P 5f705eff9b1c438c97da499bae96bb14: Deleting consensus metadata
I20260925 10:07:00.133875 4193 catalog_manager.cc:4841] TS 5f705eff9b1c438c97da499bae96bb14 (127.19.46.195:42101): tablet 20e57653b0c44e3f8af37fbbdffb1bec (table test-table [id=6c22a0ea81364ff3a809e0d6dc5c6472]) successfully deleted
W20260925 10:07:00.136633 4191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:44515 (error 108)
I20260925 10:07:00.136969 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4402
I20260925 10:07:00.137914 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.137199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44515 (local address 127.0.0.1:33282)
0925 10:07:00.137820 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.137870 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592}
W20260925 10:07:00.138123 4191 catalog_manager.cc:4584] TS d32a97c7bc784af4b1e8ea15a45c9454 (127.19.46.193:44515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20e57653b0c44e3f8af37fbbdffb1bec: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44515: connect: Connection refused (error 111)
I20260925 10:07:00.143101 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4536
I20260925 10:07:00.148087 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4174
2026-09-25T10:07:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2217 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:00Z Disabled control of system clock
I20260925 10:07:00.178778 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46479
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:00.270145 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:00.270404 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:00.270438 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:00.272118 4713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:00.272204 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:00.272218 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:00.272257 4713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:00.272270 4713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:00.274367 4713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:00.274636 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:00.274919 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:00.280367 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.280386 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.280638 4713 server_base.cc:1034] running on GCE node
W20260925 10:07:00.280367 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.281013 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:00.281353 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:00.282586 4713 hybrid_clock.cc:648] HybridClock initialized: now 1790330820282523 us; error 73 us; skew 500 ppm
I20260925 10:07:00.284627 4713 webserver.cc:460] Webserver started at http://127.19.46.254:41653/ using document root <none> and password file <none>
I20260925 10:07:00.284919 4713 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:00.284978 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:00.285082 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:00.286839 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "5d307a5ed2d94182a545a79628413744"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "b819c39db38fcb980c0c3d17d2fbf224"
server_key_iv: "c9ca41a06a9e7d0ee1cdb987a377f06e"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.287218 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "5d307a5ed2d94182a545a79628413744"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "b819c39db38fcb980c0c3d17d2fbf224"
server_key_iv: "c9ca41a06a9e7d0ee1cdb987a377f06e"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.288738 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:00.289849 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.290071 4713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:00.290180 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "5d307a5ed2d94182a545a79628413744"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "b819c39db38fcb980c0c3d17d2fbf224"
server_key_iv: "c9ca41a06a9e7d0ee1cdb987a377f06e"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.290249 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:00.307818 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:00.308161 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:00.308279 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:00.313297 4713 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46479
I20260925 10:07:00.313360 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46479 every 8 connection(s)
I20260925 10:07:00.313781 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:00.314263 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4713
I20260925 10:07:00.314384 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:00.314592 19643 external_mini_cluster.cc:1428] Setting key 9233e9b799a5e1b22626173df8d1d80e
I20260925 10:07:00.315622 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:00.319211 4780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: Bootstrap starting.
I20260925 10:07:00.319986 4780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.320322 4780 log.cc:824] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.321166 4780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: No bootstrap required, opened a new log
I20260925 10:07:00.322430 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.315716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.19.46.254:46479)
0925 10:07:00.315972 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.315981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.316014 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:07:00.316586 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.316595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.316683 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.316846 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:00.317039 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.318441 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.319317 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.319943 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.320083 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.321152 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:00.321161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:00.321161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:00.321773 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:00.321821 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:00.321836 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:00.321942 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.322104 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.322218 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260925 10:07:00.323617 4780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } }
I20260925 10:07:00.323797 4780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.323844 4780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d307a5ed2d94182a545a79628413744, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.323984 4780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } }
I20260925 10:07:00.324069 4780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:00.324115 4780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:00.324163 4780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.325101 4780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } }
I20260925 10:07:00.325254 4780 leader_election.cc:302] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d307a5ed2d94182a545a79628413744; no voters:
I20260925 10:07:00.325501 4780 leader_election.cc:288] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:00.325588 4785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:00.325742 4785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [term 1 LEADER]: Becoming Leader. State: Replica: 5d307a5ed2d94182a545a79628413744, State: Running, Role: LEADER
I20260925 10:07:00.325845 4780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:00.325835 4785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } }
I20260925 10:07:00.326241 4785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d307a5ed2d94182a545a79628413744. Latest consensus state: current_term: 1 leader_uuid: "5d307a5ed2d94182a545a79628413744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } } }
I20260925 10:07:00.326262 4786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d307a5ed2d94182a545a79628413744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d307a5ed2d94182a545a79628413744" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46479 } } }
I20260925 10:07:00.326324 4785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:00.326923 4786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:00.327119 4796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:00.327674 4796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:00.329746 4796 catalog_manager.cc:1347] Generated new cluster ID: 796577fcb28a48f78e9cbceb4cd41a12
I20260925 10:07:00.329810 4796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:00.338603 4796 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:00.339361 4796 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:00.348733 4796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: Generated new TSK 0
I20260925 10:07:00.348997 4796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:00.351107 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--builtin_ntp_servers=127.19.46.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:00.446485 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:00.446740 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:00.446885 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:00.448477 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:00.448539 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:00.448570 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:07:00.451166 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:00.451557 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:00.451921 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:00.458848 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.458848 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.459338 4804 server_base.cc:1034] running on GCE node
W20260925 10:07:00.458876 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.459803 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:00.460189 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:00.461517 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790330820461483 us; error 45 us; skew 500 ppm
I20260925 10:07:00.462970 4804 webserver.cc:460] Webserver started at http://127.19.46.193:33353/ using document root <none> and password file <none>
I20260925 10:07:00.463259 4804 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:00.463339 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:00.463498 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:00.464799 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "e90edfd18a8f41fbb4a40a8368d93377"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "51e7537a12e5c1e0c09298c659008251"
server_key_iv: "70d916078d8ae9dc76bcc4e18feaf442"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.465595 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "e90edfd18a8f41fbb4a40a8368d93377"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "51e7537a12e5c1e0c09298c659008251"
server_key_iv: "70d916078d8ae9dc76bcc4e18feaf442"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.468247 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:07:00.469497 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.469741 4804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:00.469878 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "e90edfd18a8f41fbb4a40a8368d93377"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "51e7537a12e5c1e0c09298c659008251"
server_key_iv: "70d916078d8ae9dc76bcc4e18feaf442"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.469976 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:00.518126 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:00.518509 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:00.518673 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:00.518959 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:00.519348 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:00.519412 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.519649 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:00.520035 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.527472 4804 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:44671
I20260925 10:07:00.527655 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:44671 every 8 connection(s)
I20260925 10:07:00.528036 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:07:00.535979 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.529379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:46151 (local address 127.19.46.254:46479)
0925 10:07:00.529583 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.529586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.529847 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:07:00.530918 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.530920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.530948 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.531028 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:00.532120 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.532667 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.533846 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.534063 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.534158 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.535299 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:00.535301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:00.535301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:00.535315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:00.535331 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:00.535335 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:00.535402 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.535698 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.535796 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260925 10:07:00.536175 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.529248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46479 (local address 127.19.46.193:46151)
0925 10:07:00.529684 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.529727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.530338 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.531148 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.531154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.531253 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:00.531884 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.531894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.532793 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.532794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.533728 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.533731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.534171 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.534172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.534326 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.535093 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:00.535093 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:00.535128 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:00.535155 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:00.535406 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:00.535413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:00.535416 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.535665 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.535861 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":197,"threads_started":1}
I20260925 10:07:00.537125 4932 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46479
I20260925 10:07:00.537254 4932 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:00.537631 4932 heartbeater.cc:511] Master 127.19.46.254:46479 requested a full tablet report, sending...
I20260925 10:07:00.538375 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4804
I20260925 10:07:00.538486 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:00.538475 4743 ts_manager.cc:194] Registered new tserver with Master: e90edfd18a8f41fbb4a40a8368d93377 (127.19.46.193:44671)
I20260925 10:07:00.538908 19643 external_mini_cluster.cc:1428] Setting key 7bcd795038cfebcaeab8b2ec732aa87b
I20260925 10:07:00.539817 4743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.193:46151
I20260925 10:07:00.540550 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--builtin_ntp_servers=127.19.46.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:00.630841 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:00.631107 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:00.631135 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:00.632601 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:00.632666 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:00.632700 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:07:00.634416 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:00.634766 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:00.635138 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:00.641314 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.641316 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.641574 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.641789 4935 server_base.cc:1034] running on GCE node
I20260925 10:07:00.642027 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:00.642314 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:00.643488 4935 hybrid_clock.cc:648] HybridClock initialized: now 1790330820643457 us; error 54 us; skew 500 ppm
I20260925 10:07:00.645236 4935 webserver.cc:460] Webserver started at http://127.19.46.194:39645/ using document root <none> and password file <none>
I20260925 10:07:00.645588 4935 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:00.645668 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:00.645823 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:00.647254 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "ff2291c8631444ebbc5f7bdf47ad8a51"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "0eed9a22e5aae45a91593b8b6d821111"
server_key_iv: "7c19f92dedb49b52beaf7c9cb1f1d22a"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.647785 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "ff2291c8631444ebbc5f7bdf47ad8a51"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "0eed9a22e5aae45a91593b8b6d821111"
server_key_iv: "7c19f92dedb49b52beaf7c9cb1f1d22a"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.650149 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260925 10:07:00.651201 4949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.651467 4935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:00.651615 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "ff2291c8631444ebbc5f7bdf47ad8a51"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "0eed9a22e5aae45a91593b8b6d821111"
server_key_iv: "7c19f92dedb49b52beaf7c9cb1f1d22a"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.651743 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:00.665030 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:00.665517 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:00.665715 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:00.666007 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:00.666456 4935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:00.666517 4935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.666565 4935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:00.666595 4935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.673700 4935 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:36683
I20260925 10:07:00.673790 5062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:36683 every 8 connection(s)
I20260925 10:07:00.674202 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:00.676700 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 4935
I20260925 10:07:00.676846 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:00.677079 19643 external_mini_cluster.cc:1428] Setting key 24c7b008cf80ce70bb7311a147a83b3b
I20260925 10:07:00.678390 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--builtin_ntp_servers=127.19.46.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:07:00.680825 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.675069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:53529 (local address 127.19.46.254:46479)
0925 10:07:00.675143 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.675174 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.675390 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:07:00.675952 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.675953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.675967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.676035 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:00.676849 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.677767 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.679103 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.679319 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.679422 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.680493 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:00.680495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:00.680496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:00.680501 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:00.680517 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:00.680521 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:00.680579 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.680729 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.680757 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:07:00.681113 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.675041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46479 (local address 127.19.46.194:53529)
0925 10:07:00.675268 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.675304 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.675837 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.676154 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.676157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.676165 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:00.676738 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.676743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.677888 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.677889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.678991 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.678995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.679436 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.679437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.679624 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.680292 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:00.680293 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:00.680338 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:00.680371 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:00.680603 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:00.680612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:00.680617 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.680699 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.680825 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260925 10:07:00.681747 5063 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46479
I20260925 10:07:00.681872 5063 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:00.682116 5063 heartbeater.cc:511] Master 127.19.46.254:46479 requested a full tablet report, sending...
I20260925 10:07:00.682674 4743 ts_manager.cc:194] Registered new tserver with Master: ff2291c8631444ebbc5f7bdf47ad8a51 (127.19.46.194:36683)
I20260925 10:07:00.683439 4743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.194:53529
W20260925 10:07:00.767483 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:00.767737 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:00.767763 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:00.769554 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:00.769610 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:00.769640 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:07:00.771085 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:00.771312 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:00.771591 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:00.777782 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.777819 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:00.777819 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:00.778082 5066 server_base.cc:1034] running on GCE node
I20260925 10:07:00.778324 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:00.778609 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:00.779803 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790330820779790 us; error 31 us; skew 500 ppm
I20260925 10:07:00.781572 5066 webserver.cc:460] Webserver started at http://127.19.46.195:45255/ using document root <none> and password file <none>
I20260925 10:07:00.781879 5066 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:00.781955 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:00.782127 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:00.784001 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "08b771d5b2784bc8ae0913861d501012"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "35b3f1d791454b2a8a952344acda2eaa"
server_key_iv: "99c1f6a0b09b97fec07207a049fbaa5b"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.784456 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "08b771d5b2784bc8ae0913861d501012"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "35b3f1d791454b2a8a952344acda2eaa"
server_key_iv: "99c1f6a0b09b97fec07207a049fbaa5b"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.786105 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 10:07:00.787257 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.787523 5066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:00.787647 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "08b771d5b2784bc8ae0913861d501012"
format_stamp: "Formatted at 2026-09-25 10:07:00 on dist-test-slave-f9rl"
server_key: "35b3f1d791454b2a8a952344acda2eaa"
server_key_iv: "99c1f6a0b09b97fec07207a049fbaa5b"
server_key_version: "encryptionkey@0"
I20260925 10:07:00.787753 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:00.797590 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:00.797948 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:00.798125 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:00.798389 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:00.798756 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:00.798817 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.798863 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:00.798897 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:00.805778 5066 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:33431
I20260925 10:07:00.805856 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:33431 every 8 connection(s)
I20260925 10:07:00.806289 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:00.812660 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.807263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:38525 (local address 127.19.46.254:46479)
0925 10:07:00.807360 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.807363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.807551 (+ 188us) server_negotiation.cc:407] Connection header received
0925 10:07:00.808236 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.808238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.808250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.808314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:00.809179 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.809819 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.810946 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.811169 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.811269 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.812286 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:00.812287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:00.812288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:00.812294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:00.812316 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:00.812321 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:00.812386 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.812539 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.812569 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:07:00.812915 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.807131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:46479 (local address 127.19.46.195:38525)
0925 10:07:00.807402 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.807456 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.808111 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.808395 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.808400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.808418 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:00.809004 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.809009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.809929 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.809931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.810835 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.810838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.811282 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.811284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.811443 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.812113 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:00.812114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:00.812144 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:00.812170 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:00.812446 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:00.812451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:00.812454 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.812539 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.812655 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260925 10:07:00.813558 5194 heartbeater.cc:347] Connected to a master server at 127.19.46.254:46479
I20260925 10:07:00.813730 5194 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:00.813901 5194 heartbeater.cc:511] Master 127.19.46.254:46479 requested a full tablet report, sending...
I20260925 10:07:00.814469 4743 ts_manager.cc:194] Registered new tserver with Master: 08b771d5b2784bc8ae0913861d501012 (127.19.46.195:33431)
I20260925 10:07:00.814874 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5066
I20260925 10:07:00.814961 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:00.815231 19643 external_mini_cluster.cc:1428] Setting key 1f99dbfdbb6f6100a0bf096e86f00480
I20260925 10:07:00.815237 4743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.195:38525
I20260925 10:07:00.816203 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:07:00.821146 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.19.46.254:46479)
0925 10:07:00.817740 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.817743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.817842 (+ 99us) server_negotiation.cc:407] Connection header received
0925 10:07:00.817897 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.817898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.817906 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.817959 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:00.818136 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.818612 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.819512 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.819716 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.819791 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.820734 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:00.820735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:00.820736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:00.820740 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:00.820755 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:00.820759 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:00.820814 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.820870 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.820893 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:07:00.823506 4743 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:07:00.825377 4743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:07:00.833143 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44671 (local address 127.0.0.1:42958)
0925 10:07:00.828946 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.828955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.829013 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.829302 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.829309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.829312 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.829429 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.829454 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.830811 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.830813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.831885 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.831888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.832764 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.832765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.832875 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.832893 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.832979 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.833041 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":12,"thread_start_us":56,"threads_started":1}
I20260925 10:07:00.833156 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.19.46.193:44671)
0925 10:07:00.828835 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.828838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.829081 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:07:00.829134 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.829137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.829154 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.829233 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.829610 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.830690 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.831998 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.832659 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.832762 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.832871 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.832996 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.833040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260925 10:07:00.834074 4866 tablet_service.cc:1467] Processing CreateTablet for tablet e90f00cc19c44bf78aece700ae44f7f4 (DEFAULT_TABLE table=test-table [id=9e5c7217072d4b11ae7a3b06719cc036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:00.834491 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36683 (local address 127.0.0.1:44946)
0925 10:07:00.828663 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.828713 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.828829 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.829304 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.829309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.829314 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.829429 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.829454 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.832144 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.832145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.833198 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.833200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.834198 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.834199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.834274 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.834280 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.834352 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.834401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":31,"thread_start_us":55,"threads_started":1}
I20260925 10:07:00.834589 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f00cc19c44bf78aece700ae44f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:00.834794 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.19.46.195:33431)
0925 10:07:00.828660 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.828665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.828940 (+ 275us) server_negotiation.cc:407] Connection header received
0925 10:07:00.829031 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.829033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.829052 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.829207 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.829526 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.830708 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.833130 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.834282 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.834329 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.834418 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.834652 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.834697 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260925 10:07:00.834844 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.19.46.194:36683)
0925 10:07:00.829088 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.829094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.829104 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:00.829144 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.829147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.829162 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.829238 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.830812 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.832031 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.833296 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.834087 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.834173 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.834270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.834461 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.834509 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":111,"threads_started":1}
I20260925 10:07:00.835114 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.828289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:33431 (local address 127.0.0.1:39218)
0925 10:07:00.828848 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.828859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.828940 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.829311 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.829312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.829315 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.829393 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.829395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.830811 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.830813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.832980 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.832983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.834488 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.834488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.834550 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.834554 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.834959 (+ 405us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.835027 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":991,"thread_start_us":56,"threads_started":1}
I20260925 10:07:00.835533 4997 tablet_service.cc:1467] Processing CreateTablet for tablet e90f00cc19c44bf78aece700ae44f7f4 (DEFAULT_TABLE table=test-table [id=9e5c7217072d4b11ae7a3b06719cc036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:00.835886 4997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f00cc19c44bf78aece700ae44f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:00.835878 5128 tablet_service.cc:1467] Processing CreateTablet for tablet e90f00cc19c44bf78aece700ae44f7f4 (DEFAULT_TABLE table=test-table [id=9e5c7217072d4b11ae7a3b06719cc036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:00.836267 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f00cc19c44bf78aece700ae44f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:00.838754 5208 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: Bootstrap starting.
I20260925 10:07:00.839160 5209 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Bootstrap starting.
I20260925 10:07:00.839599 5208 tablet_bootstrap.cc:654] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.839807 5209 tablet_bootstrap.cc:654] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.840128 5209 log.cc:824] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.840129 5208 log.cc:824] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.840462 5210 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Bootstrap starting.
I20260925 10:07:00.841055 5209 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: No bootstrap required, opened a new log
I20260925 10:07:00.841132 5210 tablet_bootstrap.cc:654] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:00.841140 5209 ts_tablet_manager.cc:1397] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:07:00.841217 5208 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: No bootstrap required, opened a new log
I20260925 10:07:00.841295 5208 ts_tablet_manager.cc:1397] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:07:00.841472 5210 log.cc:824] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:00.842757 5210 tablet_bootstrap.cc:492] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: No bootstrap required, opened a new log
I20260925 10:07:00.842921 5210 ts_tablet_manager.cc:1397] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:07:00.844292 5208 raft_consensus.cc:348] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.844493 5208 raft_consensus.cc:374] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.844552 5208 raft_consensus.cc:729] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90edfd18a8f41fbb4a40a8368d93377, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.844678 5209 raft_consensus.cc:348] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.844848 5208 consensus_queue.cc:260] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.845106 5209 raft_consensus.cc:374] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.845182 5209 raft_consensus.cc:729] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2291c8631444ebbc5f7bdf47ad8a51, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.845355 5208 ts_tablet_manager.cc:1428] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 10:07:00.845336 5209 consensus_queue.cc:260] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.845566 4932 heartbeater.cc:503] Master 127.19.46.254:46479 was elected leader, sending a full tablet report...
I20260925 10:07:00.845759 5209 ts_tablet_manager.cc:1428] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:07:00.845824 5063 heartbeater.cc:503] Master 127.19.46.254:46479 was elected leader, sending a full tablet report...
I20260925 10:07:00.846745 5210 raft_consensus.cc:348] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.846957 5210 raft_consensus.cc:374] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:00.846997 5210 raft_consensus.cc:729] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b771d5b2784bc8ae0913861d501012, State: Initialized, Role: FOLLOWER
I20260925 10:07:00.847160 5210 consensus_queue.cc:260] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.847488 5210 ts_tablet_manager.cc:1428] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:07:00.847514 5194 heartbeater.cc:503] Master 127.19.46.254:46479 was elected leader, sending a full tablet report...
I20260925 10:07:00.922863 5215 raft_consensus.cc:482] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:00.923095 5215 raft_consensus.cc:504] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.923537 5215 leader_election.cc:288] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b771d5b2784bc8ae0913861d501012 (127.19.46.195:33431), e90edfd18a8f41fbb4a40a8368d93377 (127.19.46.193:44671)
W20260925 10:07:00.924851 5064 tablet.cc:2369] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:00.927294 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.923748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44671 (local address 127.19.46.194:38593)
0925 10:07:00.923993 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.924003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.924073 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.924265 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.924266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.924269 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.924349 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.924350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.925273 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.925274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.926262 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.926265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.927025 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.927026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.927091 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.927094 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.927134 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.927169 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":62,"thread_start_us":134,"threads_started":1}
I20260925 10:07:00.927313 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.923853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:38593 (local address 127.19.46.193:44671)
0925 10:07:00.923899 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.923901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.924089 (+ 188us) server_negotiation.cc:407] Connection header received
0925 10:07:00.924144 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.924145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.924156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.924208 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.924515 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.925169 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.926410 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.926921 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.926966 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.926972 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.927174 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.927232 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:07:00.927367 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.923819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:58085 (local address 127.19.46.195:33431)
0925 10:07:00.923882 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.923885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.923898 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:07:00.923940 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.923941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.923953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.924027 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:00.924263 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.925051 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.926365 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.926927 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.927022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.927031 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.927237 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.927285 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:07:00.927381 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.923739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:33431 (local address 127.19.46.194:58085)
0925 10:07:00.923816 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.923829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:00.923884 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:00.924042 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:00.924043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:00.924045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:00.924141 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.924143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.925165 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.925166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.926256 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:00.926259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.927022 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.927023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:00.927138 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.927143 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:00.927210 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:07:00.927256 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":147}
I20260925 10:07:00.927754 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f00cc19c44bf78aece700ae44f7f4" candidate_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b771d5b2784bc8ae0913861d501012" is_pre_election: true
I20260925 10:07:00.927798 4886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f00cc19c44bf78aece700ae44f7f4" candidate_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90edfd18a8f41fbb4a40a8368d93377" is_pre_election: true
I20260925 10:07:00.927938 5148 raft_consensus.cc:2454] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2291c8631444ebbc5f7bdf47ad8a51 in term 0.
I20260925 10:07:00.927989 4886 raft_consensus.cc:2454] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2291c8631444ebbc5f7bdf47ad8a51 in term 0.
I20260925 10:07:00.928226 4953 leader_election.cc:302] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b771d5b2784bc8ae0913861d501012, ff2291c8631444ebbc5f7bdf47ad8a51; no voters:
I20260925 10:07:00.928409 5215 raft_consensus.cc:2781] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:00.928481 5215 raft_consensus.cc:482] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:00.928505 5215 raft_consensus.cc:3037] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.929291 5215 raft_consensus.cc:504] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.929512 5215 leader_election.cc:288] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [CANDIDATE]: Term 1 election: Requested vote from peers 08b771d5b2784bc8ae0913861d501012 (127.19.46.195:33431), e90edfd18a8f41fbb4a40a8368d93377 (127.19.46.193:44671)
I20260925 10:07:00.929672 4886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f00cc19c44bf78aece700ae44f7f4" candidate_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90edfd18a8f41fbb4a40a8368d93377"
I20260925 10:07:00.929756 4886 raft_consensus.cc:3037] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.930014 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f00cc19c44bf78aece700ae44f7f4" candidate_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b771d5b2784bc8ae0913861d501012"
I20260925 10:07:00.930138 5148 raft_consensus.cc:3037] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:00.930748 4886 raft_consensus.cc:2454] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2291c8631444ebbc5f7bdf47ad8a51 in term 1.
I20260925 10:07:00.931071 4952 leader_election.cc:302] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e90edfd18a8f41fbb4a40a8368d93377, ff2291c8631444ebbc5f7bdf47ad8a51; no voters:
I20260925 10:07:00.931123 5148 raft_consensus.cc:2454] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2291c8631444ebbc5f7bdf47ad8a51 in term 1.
I20260925 10:07:00.931393 5215 raft_consensus.cc:2781] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:00.931638 5215 raft_consensus.cc:686] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 1 LEADER]: Becoming Leader. State: Replica: ff2291c8631444ebbc5f7bdf47ad8a51, State: Running, Role: LEADER
I20260925 10:07:00.931771 5215 consensus_queue.cc:237] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } }
I20260925 10:07:00.932783 4742 catalog_manager.cc:5495] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2291c8631444ebbc5f7bdf47ad8a51 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2291c8631444ebbc5f7bdf47ad8a51" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 36683 } health_report { overall_health: HEALTHY } } }
I20260925 10:07:00.958257 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.954956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.19.46.194:36683)
0925 10:07:00.955151 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:00.955153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:00.955213 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:07:00.955356 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:00.955356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:00.955382 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:00.955484 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:00.955658 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.956216 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.957110 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:00.957383 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:00.957530 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:00.957733 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:00.957895 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:00.958020 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:00.958086 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:07:00.958115 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260925 10:07:00.961813 4886 raft_consensus.cc:1261] T e90f00cc19c44bf78aece700ae44f7f4 P e90edfd18a8f41fbb4a40a8368d93377 [term 1 FOLLOWER]: Refusing update from remote peer ff2291c8631444ebbc5f7bdf47ad8a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:00.961813 5148 raft_consensus.cc:1261] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 1 FOLLOWER]: Refusing update from remote peer ff2291c8631444ebbc5f7bdf47ad8a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:00.962700 5215 consensus_queue.cc:1035] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b771d5b2784bc8ae0913861d501012" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:00.962890 5218 consensus_queue.cc:1035] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90edfd18a8f41fbb4a40a8368d93377" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:00.966264 5221 mvcc.cc:205] Tried to move back new op lower bound from 7333195042653618176 to 7333195042537222144. Current Snapshot: MvccSnapshot[applied={T|T < 7333195042653618176}]
I20260925 10:07:00.966620 5223 mvcc.cc:205] Tried to move back new op lower bound from 7333195042653618176 to 7333195042537222144. Current Snapshot: MvccSnapshot[applied={T|T < 7333195042653618176}]
I20260925 10:07:00.967581 5222 mvcc.cc:205] Tried to move back new op lower bound from 7333195042653618176 to 7333195042537222144. Current Snapshot: MvccSnapshot[applied={T|T < 7333195042653618176}]
I20260925 10:07:00.969513 4742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:00.969626 4742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:00.970548 4742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d307a5ed2d94182a545a79628413744: Sending DeleteTablet for 3 replicas of tablet e90f00cc19c44bf78aece700ae44f7f4
I20260925 10:07:00.970942 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet e90f00cc19c44bf78aece700ae44f7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:00 UTC) from {username='slave'} at 127.0.0.1:39218
I20260925 10:07:00.970964 4997 tablet_service.cc:1514] Processing DeleteTablet for tablet e90f00cc19c44bf78aece700ae44f7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:00 UTC) from {username='slave'} at 127.0.0.1:44946
I20260925 10:07:00.971210 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4804
I20260925 10:07:00.971376 5231 tablet_replica.cc:330] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51: stopping tablet replica
I20260925 10:07:00.971501 5230 tablet_replica.cc:330] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: stopping tablet replica
I20260925 10:07:00.972375 5230 raft_consensus.cc:2229] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:00.972525 5230 raft_consensus.cc:2258] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:00.972707 5231 raft_consensus.cc:2229] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:00.972872 5231 raft_consensus.cc:2258] T e90f00cc19c44bf78aece700ae44f7f4 P ff2291c8631444ebbc5f7bdf47ad8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:00.973428 5230 ts_tablet_manager.cc:1905] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:00.976444 5230 ts_tablet_manager.cc:1918] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:00.976567 5230 log.cc:1196] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/e90f00cc19c44bf78aece700ae44f7f4
I20260925 10:07:00.977326 5230 ts_tablet_manager.cc:1939] T e90f00cc19c44bf78aece700ae44f7f4 P 08b771d5b2784bc8ae0913861d501012: Deleting consensus metadata
W20260925 10:07:00.977807 4730 connection.cc:535] client connection to 127.19.46.193:44671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:07:00.977937 4731 catalog_manager.cc:4841] TS 08b771d5b2784bc8ae0913861d501012 (127.19.46.195:33431): tablet e90f00cc19c44bf78aece700ae44f7f4 (table test-table [id=9e5c7217072d4b11ae7a3b06719cc036]) successfully deleted
W20260925 10:07:00.977946 4730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:07:00.978446 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4935
I20260925 10:07:00.978597 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.978439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44671 (local address 127.0.0.1:42962)
0925 10:07:00.978507 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.978556 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 10:07:00.979712 4730 catalog_manager.cc:4584] TS e90edfd18a8f41fbb4a40a8368d93377 (127.19.46.193:44671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e90f00cc19c44bf78aece700ae44f7f4: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44671: connect: Connection refused (error 111)
I20260925 10:07:00.984735 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5066
I20260925 10:07:00.984799 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:00.984637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:36683 (local address 127.0.0.1:44956)
0925 10:07:00.984717 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:00.984756 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.194:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:07:00.990752 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 4713
2026-09-25T10:07:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (842 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:01 dist-test-slave-f9rl krb5kdc[5235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:01 dist-test-slave-f9rl krb5kdc[5235](info): set up 2 sockets
Sep 25 10:07:01 dist-test-slave-f9rl krb5kdc[5235](info): commencing operation
krb5kdc: starting...
W20260925 10:07:02.136591 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:02.286978 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42953
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:02.383656 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:02.383922 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:02.383952 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:02.385751 5248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:02.385826 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:02.385844 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:02.385859 5248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:02.385876 5248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:02.387864 5248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:02.388150 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:02.388367 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:02.394447 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.394447 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.394558 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:02.395762 5248 server_base.cc:1034] running on GCE node
I20260925 10:07:02.396044 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:02.396342 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:02.397593 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790330822397533 us; error 76 us; skew 500 ppm
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:02.401149 5248 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:02.401811 5248 webserver.cc:460] Webserver started at http://127.19.46.254:42135/ using document root <none> and password file <none>
I20260925 10:07:02.402086 5248 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:02.402168 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:02.402339 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:02.403609 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "931b44373af84471a8810c9ade9775a7"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "492336aa7d53dae681fbe795d2982535"
server_key_iv: "63f681bd80df48c7e97d7974c27921ad"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.404053 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "931b44373af84471a8810c9ade9775a7"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "492336aa7d53dae681fbe795d2982535"
server_key_iv: "63f681bd80df48c7e97d7974c27921ad"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.405892 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:07:02.407503 5263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.407790 5248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:02.407920 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "931b44373af84471a8810c9ade9775a7"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "492336aa7d53dae681fbe795d2982535"
server_key_iv: "63f681bd80df48c7e97d7974c27921ad"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.408039 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:02.419260 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:02.422293 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:02.422569 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:02.427587 5248 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:42953
I20260925 10:07:02.427631 5315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:42953 every 8 connection(s)
I20260925 10:07:02.428038 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:02.428912 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:02.431674 5316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: Bootstrap starting.
I20260925 10:07:02.432312 5316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:02.432613 5316 log.cc:824] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:02.433562 5316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: No bootstrap required, opened a new log
I20260925 10:07:02.433777 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5248
I20260925 10:07:02.433916 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:02.434146 19643 external_mini_cluster.cc:1428] Setting key 63091c805779f0ccabd1cdbff8b20f1f
I20260925 10:07:02.435923 5316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } }
I20260925 10:07:02.436161 5316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:02.436208 5316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931b44373af84471a8810c9ade9775a7, State: Initialized, Role: FOLLOWER
I20260925 10:07:02.436364 5316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } }
I20260925 10:07:02.436457 5316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:02.436510 5316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:02.436571 5316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:02.437813 5316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } }
I20260925 10:07:02.437920 5316 leader_election.cc:302] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931b44373af84471a8810c9ade9775a7; no voters:
I20260925 10:07:02.438154 5316 leader_election.cc:288] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:02.438180 5321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:02.438302 5321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [term 1 LEADER]: Becoming Leader. State: Replica: 931b44373af84471a8810c9ade9775a7, State: Running, Role: LEADER
I20260925 10:07:02.438397 5321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } }
I20260925 10:07:02.438473 5316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:02.438783 5323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931b44373af84471a8810c9ade9775a7. Latest consensus state: current_term: 1 leader_uuid: "931b44373af84471a8810c9ade9775a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } } }
I20260925 10:07:02.438908 5323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:02.439134 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931b44373af84471a8810c9ade9775a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931b44373af84471a8810c9ade9775a7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42953 } } }
I20260925 10:07:02.439221 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7 [sys.catalog]: This master's current role is: LEADER
W20260925 10:07:02.440557 5337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:02.440623 5337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:02.440685 5327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:02.440855 5327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:02.442782 5327 catalog_manager.cc:1347] Generated new cluster ID: 334de0960359495fab858a82125d3b0e
I20260925 10:07:02.442842 5327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:02.448505 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:02.434771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.19.46.254:42953)
0925 10:07:02.434947 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:02.434955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:02.434985 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:07:02.435925 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:02.435934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:02.435999 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:02.436142 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:02.437113 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.438526 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.439746 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.440518 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.440649 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:02.443826 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:02.443830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:02.443831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:02.443844 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:02.445901 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.446347 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.446350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.446351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.446485 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.446676 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.446677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.446677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.446996 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:02.447818 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:02.448004 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 10:07:02.448256 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 10:07:02.489593 5327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:02.490339 5327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:02.499859 5327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: Generated new TSK 0
I20260925 10:07:02.500193 5327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:07:02.543191 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:02.639768 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:02.640003 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:02.640031 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:02.641373 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:02.641428 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:02.641569 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:07:02.643270 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:02.643551 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:02.643837 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:02.650348 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.650348 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.650381 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:02.650381 5342 server_base.cc:1034] running on GCE node
I20260925 10:07:02.650982 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:02.651230 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:02.652746 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790330822652713 us; error 127 us; skew 500 ppm
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:02.656117 5342 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:07:02.656669 5342 webserver.cc:460] Webserver started at http://127.19.46.193:43325/ using document root <none> and password file <none>
I20260925 10:07:02.656932 5342 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:02.657007 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:02.657159 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:02.658390 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "5f5fdf2afae84b2a84883d9b6ce5397e"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "3eba1a56d33330a026299c7b968a6114"
server_key_iv: "56c45b6f73cc28b9dacb5de7fdb13cbe"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.658792 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "5f5fdf2afae84b2a84883d9b6ce5397e"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "3eba1a56d33330a026299c7b968a6114"
server_key_iv: "56c45b6f73cc28b9dacb5de7fdb13cbe"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.660338 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:07:02.661952 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.662226 5342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:02.662353 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "5f5fdf2afae84b2a84883d9b6ce5397e"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "3eba1a56d33330a026299c7b968a6114"
server_key_iv: "56c45b6f73cc28b9dacb5de7fdb13cbe"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.662525 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:02.680042 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:02.682967 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:02.683171 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:02.683456 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:02.683988 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:02.684068 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.684118 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:02.684154 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.691848 5342 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:38099
I20260925 10:07:02.691987 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:38099 every 8 connection(s)
I20260925 10:07:02.692482 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:07:02.699858 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5342
I20260925 10:07:02.700006 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:02.700245 19643 external_mini_cluster.cc:1428] Setting key 1490307cf9191a8a0c03b651bca04b3e
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:02.705644 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:02.693837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:41661 (local address 127.19.46.254:42953)
0925 10:07:02.693933 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:02.693936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:02.693996 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:07:02.694811 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:02.694812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:02.694826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:02.694881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:02.695936 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.696480 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.697977 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.698215 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.698289 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:02.701769 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:02.701771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:02.701772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:02.701783 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:02.703516 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.704165 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.704166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.704166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.704226 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.704654 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.704655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.704656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.704793 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:02.704860 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:02.705308 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 10:07:02.705493 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:07:02.705835 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:02.693615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42953 (local address 127.19.46.193:41661)
0925 10:07:02.693828 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:02.693903 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:02.694682 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:02.694975 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:02.694979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:02.695585 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:02.695791 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:02.695795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.696623 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.696625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:02.697690 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:02.697694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.698329 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.698330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:02.698454 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:02.699233 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:02.699242 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:02.701652 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:02.703626 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:02.703643 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:02.703647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:02.704062 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:02.704298 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:02.704299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:02.704299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:02.704465 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:02.704872 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:02.704880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:02.705100 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 10:07:02.705228 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:07:02.705506 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260925 10:07:02.706702 5471 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42953
I20260925 10:07:02.706851 5471 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:02.707072 5471 heartbeater.cc:511] Master 127.19.46.254:42953 requested a full tablet report, sending...
I20260925 10:07:02.707666 5280 ts_manager.cc:194] Registered new tserver with Master: 5f5fdf2afae84b2a84883d9b6ce5397e (127.19.46.193:38099)
I20260925 10:07:02.708359 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:41661
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:07:02.734876 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:02.824793 5476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:02.825062 5476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:02.825093 5476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:02.826679 5476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:02.826740 5476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:02.826774 5476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:07:02.828701 5476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:02.829072 5476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:02.829360 5476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:02.837118 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.837224 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:02.837118 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:02.837399 5476 server_base.cc:1034] running on GCE node
I20260925 10:07:02.837697 5476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:02.838187 5476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:02.839903 5476 hybrid_clock.cc:648] HybridClock initialized: now 1790330822839873 us; error 38 us; skew 500 ppm
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:02.843927 5476 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:07:02.844671 5476 webserver.cc:460] Webserver started at http://127.19.46.194:37351/ using document root <none> and password file <none>
I20260925 10:07:02.844946 5476 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:02.845029 5476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:02.845218 5476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:02.846668 5476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "db540c644ffa4f028214ce350b60704c"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "4b8195de4571fa34a00a37443ff2d57f"
server_key_iv: "abb028253bc193fe7d35c626c603b067"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.847092 5476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "db540c644ffa4f028214ce350b60704c"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "4b8195de4571fa34a00a37443ff2d57f"
server_key_iv: "abb028253bc193fe7d35c626c603b067"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.848986 5476 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:07:02.850580 5491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.851025 5476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:02.851305 5476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "db540c644ffa4f028214ce350b60704c"
format_stamp: "Formatted at 2026-09-25 10:07:02 on dist-test-slave-f9rl"
server_key: "4b8195de4571fa34a00a37443ff2d57f"
server_key_iv: "abb028253bc193fe7d35c626c603b067"
server_key_version: "encryptionkey@0"
I20260925 10:07:02.851718 5476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:02.863226 5476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:02.866473 5476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:02.867007 5476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:02.867337 5476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:02.867799 5476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:02.867844 5476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.867873 5476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:02.867888 5476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:02.877893 5476 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:44267
I20260925 10:07:02.877944 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:44267 every 8 connection(s)
I20260925 10:07:02.878345 5476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:02.881736 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5476
I20260925 10:07:02.881942 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:02.882213 19643 external_mini_cluster.cc:1428] Setting key 61abbff46f5bd01e8a201d6e15d8ff55
Sep 25 10:07:02 dist-test-slave-f9rl krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330822, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:07:02.892638 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:02.879560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:52007 (local address 127.19.46.254:42953)
0925 10:07:02.879625 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:02.879627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:02.879828 (+ 201us) server_negotiation.cc:407] Connection header received
0925 10:07:02.880525 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:02.880527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:02.880541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:02.880633 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:02.882517 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.883191 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.884479 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.884926 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.885016 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:02.887659 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:02.887662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:02.887662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:02.887673 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:02.890295 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.890954 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.890956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.890956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.891031 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:02.891629 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:02.891631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:02.891632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:02.891805 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:02.891894 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:02.892478 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 10:07:02.892562 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:07:02.893580 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:02.879334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42953 (local address 127.19.46.194:52007)
0925 10:07:02.879576 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:02.879635 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:02.880310 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:02.880728 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:02.880733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:02.881945 (+ 1212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:02.882329 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:02.882354 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.883305 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.883306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:02.884342 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:02.884346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:02.885075 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:02.885076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:02.885208 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:02.885973 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:02.885984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:02.887536 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:02.890532 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:02.890544 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:02.890546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:02.890854 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:02.891192 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:02.891194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:02.891195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:02.891497 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:02.892017 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:02.892028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:02.892267 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 10:07:02.892415 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 10:07:02.892986 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":133,"threads_started":1}
I20260925 10:07:02.894253 5606 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42953
I20260925 10:07:02.894397 5606 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:02.895215 5606 heartbeater.cc:511] Master 127.19.46.254:42953 requested a full tablet report, sending...
I20260925 10:07:02.895962 5280 ts_manager.cc:194] Registered new tserver with Master: db540c644ffa4f028214ce350b60704c (127.19.46.194:44267)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
I20260925 10:07:02.896390 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:52007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:07:02.919585 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:03.012451 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:03.012799 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:03.012830 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:03.014518 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:03.014585 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:03.014619 5611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:07:03.016410 5611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:03.016690 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:03.016902 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:03.022811 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.022818 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.022818 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:03.023128 5611 server_base.cc:1034] running on GCE node
I20260925 10:07:03.023393 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:03.023720 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:03.024971 5611 hybrid_clock.cc:648] HybridClock initialized: now 1790330823024933 us; error 77 us; skew 500 ppm
Sep 25 10:07:03 dist-test-slave-f9rl krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330823, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:03.035171 5611 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:07:03.035795 5611 webserver.cc:460] Webserver started at http://127.19.46.195:42611/ using document root <none> and password file <none>
I20260925 10:07:03.036047 5611 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:03.036116 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:03.036265 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:03.037531 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "6e16f97331dc4920916aa61fb56929f8"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "0bf465e9f69b58b821a3c3dadf83bc43"
server_key_iv: "6d570fbc9cf7a42c7962dfee960c4573"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.037902 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "6e16f97331dc4920916aa61fb56929f8"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "0bf465e9f69b58b821a3c3dadf83bc43"
server_key_iv: "6d570fbc9cf7a42c7962dfee960c4573"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.039584 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:07:03.040627 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.040922 5611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:03.041049 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "6e16f97331dc4920916aa61fb56929f8"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "0bf465e9f69b58b821a3c3dadf83bc43"
server_key_iv: "6d570fbc9cf7a42c7962dfee960c4573"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.041138 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:03.053676 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:03.056452 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:03.056676 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:03.056954 5611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:03.057363 5611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:03.057436 5611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.057554 5611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:03.057587 5611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.065021 5611 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:42959
I20260925 10:07:03.065105 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:42959 every 8 connection(s)
I20260925 10:07:03.065527 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:03.066044 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5611
I20260925 10:07:03.066172 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:03.066391 19643 external_mini_cluster.cc:1428] Setting key 21de4fc3dcb172920b89e9f0f5a99669
Sep 25 10:07:03 dist-test-slave-f9rl krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330823, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:03.076781 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.066559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:50421 (local address 127.19.46.254:42953)
0925 10:07:03.066629 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.066631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.066771 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:07:03.067495 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.067497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.067510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.067581 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.068540 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.069121 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.070312 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.070537 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.070611 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.073170 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.073171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.073172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:03.073180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.075033 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:03.075717 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:03.075718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:03.075718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:03.075784 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:03.076061 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:03.076062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:03.076063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:03.076196 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.076261 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.076615 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.076703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:07:03.077082 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.066443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42953 (local address 127.19.46.195:50421)
0925 10:07:03.066646 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.066697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.067385 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.067633 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.067637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.068228 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:03.068425 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.068429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.069242 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.069243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.070208 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.070211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.070643 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.070644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.070763 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.071315 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:03.071324 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:03.073052 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:03.075155 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:03.075164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:03.075167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:03.075554 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:03.075864 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:03.075865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:03.075865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:03.075987 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:03.076295 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:03.076302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:03.076450 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.076558 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.076764 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260925 10:07:03.077876 5740 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42953
I20260925 10:07:03.078045 5740 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:03.078264 5740 heartbeater.cc:511] Master 127.19.46.254:42953 requested a full tablet report, sending...
I20260925 10:07:03.078735 5280 ts_manager.cc:194] Registered new tserver with Master: 6e16f97331dc4920916aa61fb56929f8 (127.19.46.195:42959)
I20260925 10:07:03.079118 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:50421
I20260925 10:07:03.089732 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:07:03.102383 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.093867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.19.46.254:42953)
0925 10:07:03.094064 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.094066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.094081 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:07:03.094204 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.094205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.094220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.094303 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.095093 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.095812 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.096760 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.097058 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.097129 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.099015 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.099017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.099018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:03.099026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.100690 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:03.101179 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:03.101180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:03.101180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:03.101233 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:03.101615 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:03.101616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:03.101616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:03.101781 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.101866 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.102125 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.102243 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 10:07:03.105006 5280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:07:03.107064 5280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:07:03.115341 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38099 (local address 127.0.0.1:60494)
0925 10:07:03.110349 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.110359 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.110398 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.110904 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.110905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.110907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.110957 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.110958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.113184 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.113185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.114175 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.114178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.114939 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.114940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.115052 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.115071 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.115159 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.115228 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260925 10:07:03.115358 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.19.46.194:44267)
0925 10:07:03.110578 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.110584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.110596 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:03.110641 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.110643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.110660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.110758 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.110948 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.112126 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.113883 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.114978 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.115077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.115139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.115215 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.115266 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260925 10:07:03.115674 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.19.46.193:38099)
0925 10:07:03.110730 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.110735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.110744 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:03.110785 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.110787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.110802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.111084 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.111161 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.113057 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.114275 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.114822 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.114933 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.115002 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.115497 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.115562 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260925 10:07:03.115341 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:44267 (local address 127.0.0.1:44666)
0925 10:07:03.110256 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.110299 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.110375 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.110766 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.110770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.110773 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.110854 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.110857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.112305 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.112306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.113754 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.113758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.115078 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.115079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.115141 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.115149 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.115218 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.115251 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260925 10:07:03.116518 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c (DEFAULT_TABLE table=test-table [id=0d0b30bd590640aa9d6b22b56b3eb358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.116930 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a1b9e1a2a64203ae5f873ba325807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.117220 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:42959 (local address 127.0.0.1:51658)
0925 10:07:03.110488 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.110496 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.110600 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.111096 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.111097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.111099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.111134 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.111135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.113643 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.113645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.115075 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.115078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.116866 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.116867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.116946 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.116952 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.117074 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.117134 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":98,"threads_started":1}
I20260925 10:07:03.117254 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.110280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.19.46.195:42959)
0925 10:07:03.110885 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.110889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.110898 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:03.110937 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.110939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.110953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.111024 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.111223 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.113073 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.115186 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.116731 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.116825 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.116882 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.117099 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.117152 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":94,"threads_started":1}
I20260925 10:07:03.117576 5540 tablet_service.cc:1467] Processing CreateTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c (DEFAULT_TABLE table=test-table [id=0d0b30bd590640aa9d6b22b56b3eb358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.118112 5540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a1b9e1a2a64203ae5f873ba325807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.118412 5674 tablet_service.cc:1467] Processing CreateTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c (DEFAULT_TABLE table=test-table [id=0d0b30bd590640aa9d6b22b56b3eb358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.118875 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a1b9e1a2a64203ae5f873ba325807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.122053 5757 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: Bootstrap starting.
I20260925 10:07:03.122179 5758 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Bootstrap starting.
I20260925 10:07:03.122706 5759 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Bootstrap starting.
I20260925 10:07:03.122946 5758 tablet_bootstrap.cc:654] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.122987 5757 tablet_bootstrap.cc:654] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.123533 5757 log.cc:824] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.123554 5759 tablet_bootstrap.cc:654] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.123584 5758 log.cc:824] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.124184 5759 log.cc:824] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.124498 5758 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: No bootstrap required, opened a new log
I20260925 10:07:03.124607 5758 ts_tablet_manager.cc:1397] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:07:03.124627 5757 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: No bootstrap required, opened a new log
I20260925 10:07:03.124701 5757 ts_tablet_manager.cc:1397] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:07:03.125133 5759 tablet_bootstrap.cc:492] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: No bootstrap required, opened a new log
I20260925 10:07:03.125252 5759 ts_tablet_manager.cc:1397] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:07:03.126605 5757 raft_consensus.cc:348] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.126802 5757 raft_consensus.cc:374] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.126852 5757 raft_consensus.cc:729] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5fdf2afae84b2a84883d9b6ce5397e, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.126978 5757 consensus_queue.cc:260] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.127321 5757 ts_tablet_manager.cc:1428] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:07:03.127367 5471 heartbeater.cc:503] Master 127.19.46.254:42953 was elected leader, sending a full tablet report...
I20260925 10:07:03.127421 5759 raft_consensus.cc:348] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.127604 5759 raft_consensus.cc:374] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.127640 5759 raft_consensus.cc:729] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e16f97331dc4920916aa61fb56929f8, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.127772 5758 raft_consensus.cc:348] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.127789 5759 consensus_queue.cc:260] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.127954 5758 raft_consensus.cc:374] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.128002 5758 raft_consensus.cc:729] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db540c644ffa4f028214ce350b60704c, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.128123 5758 consensus_queue.cc:260] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.128193 5759 ts_tablet_manager.cc:1428] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:07:03.128311 5740 heartbeater.cc:503] Master 127.19.46.254:42953 was elected leader, sending a full tablet report...
I20260925 10:07:03.128433 5758 ts_tablet_manager.cc:1428] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:07:03.128479 5606 heartbeater.cc:503] Master 127.19.46.254:42953 was elected leader, sending a full tablet report...
W20260925 10:07:03.129333 5607 tablet.cc:2369] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:03.138229 5764 raft_consensus.cc:482] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:03.138381 5764 raft_consensus.cc:504] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.138845 5764 leader_election.cc:288] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db540c644ffa4f028214ce350b60704c (127.19.46.194:44267), 5f5fdf2afae84b2a84883d9b6ce5397e (127.19.46.193:38099)
I20260925 10:07:03.142391 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.139164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:53315 (local address 127.19.46.194:44267)
0925 10:07:03.139219 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.139222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.139417 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:07:03.139513 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.139514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.139533 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.139606 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.139777 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.140399 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.141496 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.141940 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.142053 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.142063 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.142242 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.142287 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:07:03.142462 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.139078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38099 (local address 127.19.46.195:42179)
0925 10:07:03.139152 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.139166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.139218 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.139399 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.139400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.139402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.139498 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.139500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.140505 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.140507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.141343 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.141345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.142080 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.142081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.142199 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.142203 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.142262 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.142321 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:07:03.142580 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.139154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:42179 (local address 127.19.46.193:38099)
0925 10:07:03.139209 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.139211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.139222 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:03.139300 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.139301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.139312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.139396 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.139581 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.140370 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.141539 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.141972 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.142082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.142092 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.142433 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.142487 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:07:03.143065 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a1b9e1a2a64203ae5f873ba325807c" candidate_uuid: "6e16f97331dc4920916aa61fb56929f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" is_pre_election: true
I20260925 10:07:03.143271 5425 raft_consensus.cc:2454] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e16f97331dc4920916aa61fb56929f8 in term 0.
I20260925 10:07:03.143517 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.139106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:44267 (local address 127.19.46.195:53315)
0925 10:07:03.139330 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.139349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.139435 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.139608 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.139609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.139611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.139700 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.139702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.140507 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.140508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.141350 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.141353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.142054 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.142056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.142146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.142150 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.142234 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.142388 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260925 10:07:03.143627 5628 leader_election.cc:302] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f5fdf2afae84b2a84883d9b6ce5397e, 6e16f97331dc4920916aa61fb56929f8; no voters:
I20260925 10:07:03.143805 5764 raft_consensus.cc:2781] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:03.143889 5764 raft_consensus.cc:482] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:03.143927 5764 raft_consensus.cc:3037] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.143930 5560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a1b9e1a2a64203ae5f873ba325807c" candidate_uuid: "6e16f97331dc4920916aa61fb56929f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db540c644ffa4f028214ce350b60704c" is_pre_election: true
I20260925 10:07:03.144191 5560 raft_consensus.cc:2454] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e16f97331dc4920916aa61fb56929f8 in term 0.
I20260925 10:07:03.144929 5764 raft_consensus.cc:504] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.145103 5764 leader_election.cc:288] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [CANDIDATE]: Term 1 election: Requested vote from peers db540c644ffa4f028214ce350b60704c (127.19.46.194:44267), 5f5fdf2afae84b2a84883d9b6ce5397e (127.19.46.193:38099)
I20260925 10:07:03.145277 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a1b9e1a2a64203ae5f873ba325807c" candidate_uuid: "6e16f97331dc4920916aa61fb56929f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e"
I20260925 10:07:03.145315 5560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a1b9e1a2a64203ae5f873ba325807c" candidate_uuid: "6e16f97331dc4920916aa61fb56929f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db540c644ffa4f028214ce350b60704c"
I20260925 10:07:03.145395 5425 raft_consensus.cc:3037] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.145524 5560 raft_consensus.cc:3037] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.146409 5425 raft_consensus.cc:2454] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e16f97331dc4920916aa61fb56929f8 in term 1.
I20260925 10:07:03.146486 5560 raft_consensus.cc:2454] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e16f97331dc4920916aa61fb56929f8 in term 1.
I20260925 10:07:03.146663 5628 leader_election.cc:302] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f5fdf2afae84b2a84883d9b6ce5397e, 6e16f97331dc4920916aa61fb56929f8; no voters:
I20260925 10:07:03.146837 5764 raft_consensus.cc:2781] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:03.147053 5764 raft_consensus.cc:686] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 1 LEADER]: Becoming Leader. State: Replica: 6e16f97331dc4920916aa61fb56929f8, State: Running, Role: LEADER
I20260925 10:07:03.147176 5764 consensus_queue.cc:237] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } }
I20260925 10:07:03.148424 5279 catalog_manager.cc:5495] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e16f97331dc4920916aa61fb56929f8 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "6e16f97331dc4920916aa61fb56929f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e16f97331dc4920916aa61fb56929f8" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 42959 } health_report { overall_health: HEALTHY } } }
I20260925 10:07:03.156988 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.154036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.19.46.195:42959)
0925 10:07:03.154130 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.154132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.154143 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:03.154189 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.154190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.154214 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.154300 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:03.154452 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.155007 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.155830 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.156207 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.156404 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.156578 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:03.156674 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:03.156804 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.156863 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.156878 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:07:03.159530 5425 raft_consensus.cc:1261] T 51a1b9e1a2a64203ae5f873ba325807c P 5f5fdf2afae84b2a84883d9b6ce5397e [term 1 FOLLOWER]: Refusing update from remote peer 6e16f97331dc4920916aa61fb56929f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:03.159651 5560 raft_consensus.cc:1261] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 1 FOLLOWER]: Refusing update from remote peer 6e16f97331dc4920916aa61fb56929f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:03.160010 5764 consensus_queue.cc:1035] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5fdf2afae84b2a84883d9b6ce5397e" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:03.160164 5767 consensus_queue.cc:1035] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db540c644ffa4f028214ce350b60704c" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:03.163990 5770 mvcc.cc:205] Tried to move back new op lower bound from 7333195051657764864 to 7333195051611860992. Current Snapshot: MvccSnapshot[applied={T|T < 7333195051657764864}]
I20260925 10:07:03.164588 5771 mvcc.cc:205] Tried to move back new op lower bound from 7333195051657764864 to 7333195051611860992. Current Snapshot: MvccSnapshot[applied={T|T < 7333195051657764864}]
I20260925 10:07:03.167294 5279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:03.167454 5279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:03.168601 5279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 931b44373af84471a8810c9ade9775a7: Sending DeleteTablet for 3 replicas of tablet 51a1b9e1a2a64203ae5f873ba325807c
I20260925 10:07:03.169060 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:60494
I20260925 10:07:03.169075 5540 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:44666
I20260925 10:07:03.169127 5674 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a1b9e1a2a64203ae5f873ba325807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:51658
I20260925 10:07:03.169368 5782 tablet_replica.cc:330] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: stopping tablet replica
I20260925 10:07:03.169567 5782 raft_consensus.cc:2229] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:03.169718 5782 raft_consensus.cc:2258] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:03.169749 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5342
I20260925 10:07:03.170423 5782 ts_tablet_manager.cc:1905] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:03.172775 5782 ts_tablet_manager.cc:1918] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:03.173097 5782 log.cc:1196] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/51a1b9e1a2a64203ae5f873ba325807c
I20260925 10:07:03.173586 5781 tablet_replica.cc:330] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: stopping tablet replica
I20260925 10:07:03.173696 5782 ts_tablet_manager.cc:1939] T 51a1b9e1a2a64203ae5f873ba325807c P db540c644ffa4f028214ce350b60704c: Deleting consensus metadata
I20260925 10:07:03.173766 5781 raft_consensus.cc:2229] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:03.173959 5781 raft_consensus.cc:2258] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:03.174461 5267 catalog_manager.cc:4841] TS db540c644ffa4f028214ce350b60704c (127.19.46.194:44267): tablet 51a1b9e1a2a64203ae5f873ba325807c (table test-table [id=0d0b30bd590640aa9d6b22b56b3eb358]) successfully deleted
I20260925 10:07:03.174912 5781 ts_tablet_manager.cc:1905] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:03.177075 5781 ts_tablet_manager.cc:1918] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:03.177390 5781 log.cc:1196] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/51a1b9e1a2a64203ae5f873ba325807c
I20260925 10:07:03.177778 5781 ts_tablet_manager.cc:1939] T 51a1b9e1a2a64203ae5f873ba325807c P 6e16f97331dc4920916aa61fb56929f8: Deleting consensus metadata
W20260925 10:07:03.178076 5265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:38099 (error 108)
I20260925 10:07:03.178277 5266 catalog_manager.cc:4841] TS 6e16f97331dc4920916aa61fb56929f8 (127.19.46.195:42959): tablet 51a1b9e1a2a64203ae5f873ba325807c (table test-table [id=0d0b30bd590640aa9d6b22b56b3eb358]) successfully deleted
I20260925 10:07:03.178732 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.178594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:38099 (local address 127.0.0.1:60502)
0925 10:07:03.178655 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.178694 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:07:03.178711 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5476
W20260925 10:07:03.179342 5265 catalog_manager.cc:4584] TS 5f5fdf2afae84b2a84883d9b6ce5397e (127.19.46.193:38099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51a1b9e1a2a64203ae5f873ba325807c: Network error: Client connection negotiation failed: client connection to 127.19.46.193:38099: connect: Connection refused (error 111)
I20260925 10:07:03.183595 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5611
I20260925 10:07:03.189175 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5248
2026-09-25T10:07:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2197 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:03Z Disabled control of system clock
I20260925 10:07:03.219772 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42903
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:03.313704 5787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:03.313975 5787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:03.314003 5787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:03.315618 5787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:03.315680 5787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:03.315693 5787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:03.315709 5787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:03.315723 5787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:03.317577 5787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:42903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:03.317936 5787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:03.318323 5787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:03.324579 5795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.324636 5793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.324636 5792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:03.324736 5787 server_base.cc:1034] running on GCE node
I20260925 10:07:03.325121 5787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:03.325546 5787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:03.326747 5787 hybrid_clock.cc:648] HybridClock initialized: now 1790330823326724 us; error 42 us; skew 500 ppm
I20260925 10:07:03.328565 5787 webserver.cc:460] Webserver started at http://127.19.46.254:33713/ using document root <none> and password file <none>
I20260925 10:07:03.328896 5787 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:03.328981 5787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:03.329160 5787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:03.330482 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "bd61d86f21e94ddcb13753609c13a2f7"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "75707e2289468b1122889a8e98096a3c"
server_key_iv: "260fb751cc800f27657048e13c7c29d6"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.330888 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "bd61d86f21e94ddcb13753609c13a2f7"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "75707e2289468b1122889a8e98096a3c"
server_key_iv: "260fb751cc800f27657048e13c7c29d6"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.332446 5787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:07:03.333670 5801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.334311 5787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:03.334415 5787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "bd61d86f21e94ddcb13753609c13a2f7"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "75707e2289468b1122889a8e98096a3c"
server_key_iv: "260fb751cc800f27657048e13c7c29d6"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.334635 5787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:03.347188 5787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:03.347589 5787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:03.347754 5787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:03.352880 5787 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:42903
I20260925 10:07:03.352922 5853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:42903 every 8 connection(s)
I20260925 10:07:03.353338 5787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:03.354583 5854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.355548 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5787
I20260925 10:07:03.355698 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:03.355911 19643 external_mini_cluster.cc:1428] Setting key 5f5a5408a36ca13b08a2b0a4b2234016
I20260925 10:07:03.357919 5854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: Bootstrap starting.
I20260925 10:07:03.358640 5854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.358970 5854 log.cc:824] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.359838 5854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: No bootstrap required, opened a new log
I20260925 10:07:03.362720 5854 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } }
I20260925 10:07:03.362963 5854 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.363015 5854 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd61d86f21e94ddcb13753609c13a2f7, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.363184 5854 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } }
I20260925 10:07:03.363279 5854 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:03.363307 5854 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:03.363384 5854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.364164 5854 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } }
I20260925 10:07:03.364274 5854 leader_election.cc:302] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd61d86f21e94ddcb13753609c13a2f7; no voters:
I20260925 10:07:03.364372 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.356754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.19.46.254:42903)
0925 10:07:03.357017 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.357233 (+ 216us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.357406 (+ 173us) server_negotiation.cc:407] Connection header received
0925 10:07:03.358224 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.358235 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.358318 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.358485 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.358845 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.360230 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.361760 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.362492 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.362682 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.363554 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.363558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.363558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:03.363747 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.363784 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:03.363796 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.363853 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.363976 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.364087 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260925 10:07:03.364498 5854 leader_election.cc:288] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:03.364598 5859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:03.364797 5859 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [term 1 LEADER]: Becoming Leader. State: Replica: bd61d86f21e94ddcb13753609c13a2f7, State: Running, Role: LEADER
I20260925 10:07:03.364804 5854 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:03.364953 5859 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } }
I20260925 10:07:03.365381 5860 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd61d86f21e94ddcb13753609c13a2f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } } }
I20260925 10:07:03.365507 5860 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:03.366305 5861 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd61d86f21e94ddcb13753609c13a2f7. Latest consensus state: current_term: 1 leader_uuid: "bd61d86f21e94ddcb13753609c13a2f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd61d86f21e94ddcb13753609c13a2f7" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42903 } } }
I20260925 10:07:03.366650 5861 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7 [sys.catalog]: This master's current role is: LEADER
W20260925 10:07:03.366845 5875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:03.366895 5875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:03.366964 5868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:03.367385 5868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:03.369927 5868 catalog_manager.cc:1347] Generated new cluster ID: b07a68b0f4fe4cee82c16108e87e61ac
I20260925 10:07:03.370010 5868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:03.385527 5868 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:03.386225 5868 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:03.390820 5868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: Generated new TSK 0
I20260925 10:07:03.391098 5868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:03.399214 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--builtin_ntp_servers=127.19.46.212:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:03.506315 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:03.506551 5878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:03.506585 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:03.507913 5878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:03.507969 5878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:03.507999 5878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:07:03.509570 5878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:03.509859 5878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:03.510226 5878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:03.515750 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.515750 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.515869 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:03.516218 5878 server_base.cc:1034] running on GCE node
I20260925 10:07:03.516500 5878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:03.516790 5878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:03.518054 5878 hybrid_clock.cc:648] HybridClock initialized: now 1790330823518020 us; error 34 us; skew 500 ppm
I20260925 10:07:03.520030 5878 webserver.cc:460] Webserver started at http://127.19.46.193:38911/ using document root <none> and password file <none>
I20260925 10:07:03.520296 5878 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:03.520345 5878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:03.520526 5878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:03.521746 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "920a41100f654ece9cd81a8e08a6991c"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "b80b9b62b01fd06b7d0f99171ce1c820"
server_key_iv: "4f075d0350656db7d4a943455bcdfc84"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.522114 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "920a41100f654ece9cd81a8e08a6991c"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "b80b9b62b01fd06b7d0f99171ce1c820"
server_key_iv: "4f075d0350656db7d4a943455bcdfc84"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.523478 5878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:03.524318 5892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.524547 5878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:03.524645 5878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "920a41100f654ece9cd81a8e08a6991c"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "b80b9b62b01fd06b7d0f99171ce1c820"
server_key_iv: "4f075d0350656db7d4a943455bcdfc84"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.524744 5878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:03.534046 5878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:03.534453 5878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:03.534633 5878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:03.534904 5878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:03.535290 5878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:03.535353 5878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.535400 5878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:03.535472 5878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.541996 5878 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:32775
I20260925 10:07:03.542073 6005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:32775 every 8 connection(s)
I20260925 10:07:03.542531 5878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:07:03.546821 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 5878
I20260925 10:07:03.546918 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:03.547159 19643 external_mini_cluster.cc:1428] Setting key 9221b1489a35fa415725b33d36cbe20a
I20260925 10:07:03.548367 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--builtin_ntp_servers=127.19.46.212:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:07:03.548897 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.543386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38697 (local address 127.19.46.254:42903)
0925 10:07:03.543434 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.543436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.543697 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:07:03.544206 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.544207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.544223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.544296 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.544979 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.545757 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.547099 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.547399 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.547492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.548620 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.548622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.548622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:03.548628 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.548646 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:03.548650 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.548704 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.548782 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.548812 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:07:03.549183 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.543382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42903 (local address 127.19.46.193:38697)
0925 10:07:03.543559 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.543598 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.544107 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.544350 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.544353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.544362 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:03.544884 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.544889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.545886 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.545888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.546963 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.546967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.547503 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.547504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.547701 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.548440 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:03.548440 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:03.548497 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:03.548536 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:03.548704 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:03.548710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:03.548714 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.548787 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.548908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260925 10:07:03.549947 6006 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42903
I20260925 10:07:03.550067 6006 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:03.550282 6006 heartbeater.cc:511] Master 127.19.46.254:42903 requested a full tablet report, sending...
I20260925 10:07:03.550822 5818 ts_manager.cc:194] Registered new tserver with Master: 920a41100f654ece9cd81a8e08a6991c (127.19.46.193:32775)
I20260925 10:07:03.551489 5818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.193:38697
W20260925 10:07:03.643677 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:03.644035 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:03.644079 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:03.646010 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:03.646124 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:03.646173 6009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:07:03.648080 6009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:03.648348 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:03.648665 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:03.654628 6017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.654637 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.654637 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:03.655298 6009 server_base.cc:1034] running on GCE node
I20260925 10:07:03.655571 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:03.655825 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:03.657102 6009 hybrid_clock.cc:648] HybridClock initialized: now 1790330823657087 us; error 51 us; skew 500 ppm
I20260925 10:07:03.659173 6009 webserver.cc:460] Webserver started at http://127.19.46.194:42165/ using document root <none> and password file <none>
I20260925 10:07:03.659546 6009 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:03.659611 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:03.659754 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:03.660918 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "6931fe9ef85f4892a0c0670a6edcd18b"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "d85c74845bc939a401f6d1fcb16a1782"
server_key_iv: "60873e1b6552f0ccc0f8a7ef6286da35"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.661299 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "6931fe9ef85f4892a0c0670a6edcd18b"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "d85c74845bc939a401f6d1fcb16a1782"
server_key_iv: "60873e1b6552f0ccc0f8a7ef6286da35"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.663158 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 10:07:03.664127 6023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.664568 6009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:03.664685 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "6931fe9ef85f4892a0c0670a6edcd18b"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "d85c74845bc939a401f6d1fcb16a1782"
server_key_iv: "60873e1b6552f0ccc0f8a7ef6286da35"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.664789 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:03.696025 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:03.696422 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:03.696593 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:03.696863 6009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:03.697419 6009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:03.697602 6009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.697652 6009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:03.697683 6009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.704516 6009 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:37641
I20260925 10:07:03.704589 6136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:37641 every 8 connection(s)
I20260925 10:07:03.705009 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:03.712070 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.706428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:46953 (local address 127.19.46.254:42903)
0925 10:07:03.706556 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.706558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.706822 (+ 264us) server_negotiation.cc:407] Connection header received
0925 10:07:03.707363 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.707364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.707377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.707522 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.708352 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.708981 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.710299 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.710599 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.710691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.711698 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.711700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.711700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:03.711707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.711728 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:03.711733 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.711797 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.711953 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.711982 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 10:07:03.712294 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.706340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42903 (local address 127.19.46.194:46953)
0925 10:07:03.706658 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.706707 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.707246 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.707630 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.707635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.707654 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:03.708232 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.708237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.709086 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.709087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.710164 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.710167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.710754 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.710755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.710917 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.711528 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:03.711529 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:03.711561 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:03.711586 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:03.711799 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:03.711806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:03.711810 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.711901 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.712027 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260925 10:07:03.712909 6137 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42903
I20260925 10:07:03.713039 6137 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:03.713238 6137 heartbeater.cc:511] Master 127.19.46.254:42903 requested a full tablet report, sending...
I20260925 10:07:03.713683 5818 ts_manager.cc:194] Registered new tserver with Master: 6931fe9ef85f4892a0c0670a6edcd18b (127.19.46.194:37641)
I20260925 10:07:03.714212 5818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.194:46953
I20260925 10:07:03.714921 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6009
I20260925 10:07:03.715050 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:03.715276 19643 external_mini_cluster.cc:1428] Setting key f2765eae71e3138e2bdcfbd69b403da8
I20260925 10:07:03.716389 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--builtin_ntp_servers=127.19.46.212:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:03.807535 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:03.807811 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:03.807854 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:03.809186 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:03.809278 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:03.809324 6140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:07:03.810956 6140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:03.811290 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:03.811641 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:03.817939 6145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.817976 6148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:03.817976 6146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:03.818296 6140 server_base.cc:1034] running on GCE node
I20260925 10:07:03.818562 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:03.819046 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:03.820253 6140 hybrid_clock.cc:648] HybridClock initialized: now 1790330823820140 us; error 144 us; skew 500 ppm
I20260925 10:07:03.822129 6140 webserver.cc:460] Webserver started at http://127.19.46.195:43575/ using document root <none> and password file <none>
I20260925 10:07:03.822695 6140 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:03.822760 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:03.822878 6140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:03.824011 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "e8c5fee42e1f472db4a9e24bc99f5172"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "f3fb52dc5b1faa6f70818411e4339509"
server_key_iv: "5c940d8be110554fcce2160229ca8c93"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.824348 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "e8c5fee42e1f472db4a9e24bc99f5172"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "f3fb52dc5b1faa6f70818411e4339509"
server_key_iv: "5c940d8be110554fcce2160229ca8c93"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.825855 6140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 10:07:03.826918 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.827279 6140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:03.827406 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "e8c5fee42e1f472db4a9e24bc99f5172"
format_stamp: "Formatted at 2026-09-25 10:07:03 on dist-test-slave-f9rl"
server_key: "f3fb52dc5b1faa6f70818411e4339509"
server_key_iv: "5c940d8be110554fcce2160229ca8c93"
server_key_version: "encryptionkey@0"
I20260925 10:07:03.827502 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:03.840330 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:03.840930 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:03.841223 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:03.841574 6140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:03.841971 6140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:03.842044 6140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.842090 6140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:03.842118 6140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:03.849238 6140 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:36051
I20260925 10:07:03.849433 6267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:36051 every 8 connection(s)
I20260925 10:07:03.850250 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:03.853739 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6140
I20260925 10:07:03.853895 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:03.854123 19643 external_mini_cluster.cc:1428] Setting key d9d178f6713580455aabae3bce19bf23
I20260925 10:07:03.858855 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.851405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55269 (local address 127.19.46.254:42903)
0925 10:07:03.851514 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.851517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.851895 (+ 378us) server_negotiation.cc:407] Connection header received
0925 10:07:03.852516 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.852518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.852531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.852637 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.853761 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.854690 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.856164 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.856639 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.856722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.858217 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.858219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.858219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:03.858226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.858253 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:03.858258 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.858333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.858696 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.858750 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:07:03.858928 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.851316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42903 (local address 127.19.46.195:55269)
0925 10:07:03.851575 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.851627 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.852348 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.852764 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.852769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.852779 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:03.853398 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.853405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.854824 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.854826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.856053 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.856057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.856801 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.856802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.856940 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.857937 (+ 997us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:03.857938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:03.858001 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:03.858050 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:03.858389 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:03.858399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:03.858403 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.858560 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.858690 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":131,"threads_started":1}
I20260925 10:07:03.859830 6268 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42903
I20260925 10:07:03.859956 6268 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:03.860227 6268 heartbeater.cc:511] Master 127.19.46.254:42903 requested a full tablet report, sending...
I20260925 10:07:03.861009 5818 ts_manager.cc:194] Registered new tserver with Master: e8c5fee42e1f472db4a9e24bc99f5172 (127.19.46.195:36051)
I20260925 10:07:03.861594 5818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.195:55269
I20260925 10:07:03.867712 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:07:03.874171 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.869817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.19.46.254:42903)
0925 10:07:03.870054 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.870056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.870068 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:03.870094 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.870094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.870106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.870214 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:03.870448 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.871051 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.871915 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.872145 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.872252 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.873737 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:03.873741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:03.873741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:03.873752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:03.873767 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:03.873771 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:03.873951 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.874054 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.874068 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260925 10:07:03.876614 5818 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:07:03.879606 5818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:07:03.887883 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.882958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:36051 (local address 127.0.0.1:52126)
0925 10:07:03.883176 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.883220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.883294 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.883665 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.883668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.883671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.883772 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.883775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885222 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.885223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.886510 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.886513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.887413 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.887415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.887529 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.887542 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.887701 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.887782 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":101,"thread_start_us":99,"threads_started":1}
I20260925 10:07:03.887893 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.883105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.19.46.195:36051)
0925 10:07:03.883456 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.883461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.883471 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:03.883532 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.883534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.883552 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.883848 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.883904 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885117 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.886609 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.887281 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.887369 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.887479 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.887733 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.887783 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":212,"threads_started":1}
I20260925 10:07:03.888219 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.883174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.19.46.193:32775)
0925 10:07:03.883375 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.883380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.883550 (+ 170us) server_negotiation.cc:407] Connection header received
0925 10:07:03.883674 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.883676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.883697 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.883747 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.883989 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885117 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.886987 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.887699 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.887771 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.887850 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.888082 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.888133 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260925 10:07:03.888854 6201 tablet_service.cc:1467] Processing CreateTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c (DEFAULT_TABLE table=test-table [id=64ec5880662f4de990f20b7e8a78feda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.889204 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.883047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:32775 (local address 127.0.0.1:52836)
0925 10:07:03.883478 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.883489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.883580 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.883869 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.883869 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.883871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.883907 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.883908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885229 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.885230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.886818 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.886826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.887904 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.887905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.887971 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.887977 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.889062 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.889124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":78,"thread_start_us":61,"threads_started":1}
I20260925 10:07:03.889302 6201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f9fc2b14de412db3caaf8914efaa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.889351 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.883370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37641 (local address 127.0.0.1:40838)
0925 10:07:03.883493 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.883504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.883628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.884105 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.884107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.884109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.884180 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.884182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885296 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.885297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.886818 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.886826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.889035 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.889036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.889129 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.889134 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.889198 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.889251 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":87,"thread_start_us":54,"threads_started":1}
I20260925 10:07:03.889927 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.883533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.19.46.194:37641)
0925 10:07:03.883790 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.883794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.883804 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:03.883955 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.883960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.883983 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.884259 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.884317 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.885133 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.887541 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.888858 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.888941 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.889023 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.889225 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.889827 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":188,"threads_started":1}
I20260925 10:07:03.890080 5940 tablet_service.cc:1467] Processing CreateTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c (DEFAULT_TABLE table=test-table [id=64ec5880662f4de990f20b7e8a78feda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.890513 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f9fc2b14de412db3caaf8914efaa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.890792 6071 tablet_service.cc:1467] Processing CreateTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c (DEFAULT_TABLE table=test-table [id=64ec5880662f4de990f20b7e8a78feda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:03.891132 6071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f9fc2b14de412db3caaf8914efaa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:03.892401 6282 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Bootstrap starting.
I20260925 10:07:03.893064 6282 tablet_bootstrap.cc:654] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.893384 6282 log.cc:824] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.894096 6282 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: No bootstrap required, opened a new log
I20260925 10:07:03.894186 6282 ts_tablet_manager.cc:1397] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:07:03.894520 6284 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: Bootstrap starting.
I20260925 10:07:03.894932 6285 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Bootstrap starting.
I20260925 10:07:03.895290 6284 tablet_bootstrap.cc:654] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.895649 6285 tablet_bootstrap.cc:654] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:03.895648 6284 log.cc:824] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.896073 6285 log.cc:824] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:03.896850 6285 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: No bootstrap required, opened a new log
I20260925 10:07:03.896955 6285 ts_tablet_manager.cc:1397] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:07:03.897007 6284 tablet_bootstrap.cc:492] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: No bootstrap required, opened a new log
I20260925 10:07:03.897132 6284 ts_tablet_manager.cc:1397] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:07:03.898077 6282 raft_consensus.cc:348] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.898321 6282 raft_consensus.cc:374] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.898358 6282 raft_consensus.cc:729] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c5fee42e1f472db4a9e24bc99f5172, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.898526 6282 consensus_queue.cc:260] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.898852 6282 ts_tablet_manager.cc:1428] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 10:07:03.899004 6268 heartbeater.cc:503] Master 127.19.46.254:42903 was elected leader, sending a full tablet report...
I20260925 10:07:03.899613 6285 raft_consensus.cc:348] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.899879 6285 raft_consensus.cc:374] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.899940 6285 raft_consensus.cc:729] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6931fe9ef85f4892a0c0670a6edcd18b, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.900092 6285 consensus_queue.cc:260] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.900421 6285 ts_tablet_manager.cc:1428] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 10:07:03.900454 6137 heartbeater.cc:503] Master 127.19.46.254:42903 was elected leader, sending a full tablet report...
I20260925 10:07:03.901160 6284 raft_consensus.cc:348] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.901340 6284 raft_consensus.cc:374] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:03.901377 6284 raft_consensus.cc:729] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920a41100f654ece9cd81a8e08a6991c, State: Initialized, Role: FOLLOWER
I20260925 10:07:03.901523 6284 consensus_queue.cc:260] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.901829 6284 ts_tablet_manager.cc:1428] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 10:07:03.901849 6006 heartbeater.cc:503] Master 127.19.46.254:42903 was elected leader, sending a full tablet report...
I20260925 10:07:03.949257 6288 raft_consensus.cc:482] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:03.949460 6288 raft_consensus.cc:504] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.950109 6288 leader_election.cc:288] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920a41100f654ece9cd81a8e08a6991c (127.19.46.193:32775), 6931fe9ef85f4892a0c0670a6edcd18b (127.19.46.194:37641)
I20260925 10:07:03.955063 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.950722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:55727 (local address 127.19.46.193:32775)
0925 10:07:03.950784 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.950786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.950792 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:07:03.950820 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.950821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.950828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.950898 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.951104 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.951759 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.953317 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.954249 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.954357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.954368 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.954880 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.954976 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:07:03.955127 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.950929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:47921 (local address 127.19.46.194:37641)
0925 10:07:03.950981 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.950983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.951199 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:07:03.951239 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.951240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.951253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.951305 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:03.951655 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.952182 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.953554 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.954362 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.954636 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.954647 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.954993 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.955047 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:07:03.955083 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.950466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:32775 (local address 127.19.46.195:55727)
0925 10:07:03.950566 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.950585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.950676 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.950902 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.950903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.950905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.950992 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.950995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.951876 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.951877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.953113 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.953116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.954486 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.954489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.954611 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.954626 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.954842 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.954926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260925 10:07:03.955093 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.950863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37641 (local address 127.19.46.195:47921)
0925 10:07:03.951110 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:03.951121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:03.951191 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:03.951388 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:03.951389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:03.951393 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:03.951508 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.951510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.952297 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.952298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.953382 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:03.953384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.954775 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.954777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:03.954876 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.954881 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:03.954956 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:07:03.954999 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":152,"threads_started":1}
W20260925 10:07:03.955674 6138 tablet.cc:2369] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:03.955884 5960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f9fc2b14de412db3caaf8914efaa5c" candidate_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920a41100f654ece9cd81a8e08a6991c" is_pre_election: true
I20260925 10:07:03.955894 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f9fc2b14de412db3caaf8914efaa5c" candidate_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" is_pre_election: true
I20260925 10:07:03.956035 5960 raft_consensus.cc:2454] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c5fee42e1f472db4a9e24bc99f5172 in term 0.
I20260925 10:07:03.956037 6091 raft_consensus.cc:2454] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c5fee42e1f472db4a9e24bc99f5172 in term 0.
I20260925 10:07:03.956347 6158 leader_election.cc:302] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6931fe9ef85f4892a0c0670a6edcd18b, e8c5fee42e1f472db4a9e24bc99f5172; no voters:
I20260925 10:07:03.956616 6288 raft_consensus.cc:2781] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:03.956768 6288 raft_consensus.cc:482] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:03.956809 6288 raft_consensus.cc:3037] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.957960 6288 raft_consensus.cc:504] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.958359 6288 leader_election.cc:288] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [CANDIDATE]: Term 1 election: Requested vote from peers 920a41100f654ece9cd81a8e08a6991c (127.19.46.193:32775), 6931fe9ef85f4892a0c0670a6edcd18b (127.19.46.194:37641)
I20260925 10:07:03.958767 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f9fc2b14de412db3caaf8914efaa5c" candidate_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6931fe9ef85f4892a0c0670a6edcd18b"
I20260925 10:07:03.958845 5960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f9fc2b14de412db3caaf8914efaa5c" candidate_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920a41100f654ece9cd81a8e08a6991c"
I20260925 10:07:03.958909 6091 raft_consensus.cc:3037] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.958981 5960 raft_consensus.cc:3037] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:03.959976 6091 raft_consensus.cc:2454] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c5fee42e1f472db4a9e24bc99f5172 in term 1.
I20260925 10:07:03.959991 5960 raft_consensus.cc:2454] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c5fee42e1f472db4a9e24bc99f5172 in term 1.
I20260925 10:07:03.960254 6158 leader_election.cc:302] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6931fe9ef85f4892a0c0670a6edcd18b, e8c5fee42e1f472db4a9e24bc99f5172; no voters:
I20260925 10:07:03.960506 6288 raft_consensus.cc:2781] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:03.960708 6288 raft_consensus.cc:686] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 1 LEADER]: Becoming Leader. State: Replica: e8c5fee42e1f472db4a9e24bc99f5172, State: Running, Role: LEADER
I20260925 10:07:03.960834 6288 consensus_queue.cc:237] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } }
I20260925 10:07:03.962105 5818 catalog_manager.cc:5495] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8c5fee42e1f472db4a9e24bc99f5172 (127.19.46.195). New cstate: current_term: 1 leader_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8c5fee42e1f472db4a9e24bc99f5172" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 36051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 } health_report { overall_health: UNKNOWN } } }
I20260925 10:07:03.986680 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:03.982974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.19.46.195:36051)
0925 10:07:03.983108 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:03.983110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:03.983120 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:03.983160 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:03.983161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:03.983185 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:03.983259 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:03.983566 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.984101 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.985085 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:03.985350 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:03.985518 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:03.985881 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:03.986026 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:03.986223 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:03.986549 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:07:03.986567 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 10:07:03.989847 5960 raft_consensus.cc:1261] T 83f9fc2b14de412db3caaf8914efaa5c P 920a41100f654ece9cd81a8e08a6991c [term 1 FOLLOWER]: Refusing update from remote peer e8c5fee42e1f472db4a9e24bc99f5172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:03.989907 6091 raft_consensus.cc:1261] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 1 FOLLOWER]: Refusing update from remote peer e8c5fee42e1f472db4a9e24bc99f5172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:03.990506 6292 consensus_queue.cc:1035] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6931fe9ef85f4892a0c0670a6edcd18b" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:03.990679 6288 consensus_queue.cc:1035] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920a41100f654ece9cd81a8e08a6991c" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:03.995486 6297 mvcc.cc:205] Tried to move back new op lower bound from 7333195055057580032 to 7333195054944038912. Current Snapshot: MvccSnapshot[applied={T|T < 7333195055057580032}]
I20260925 10:07:03.996959 5817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:03.997084 5817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:03.998260 5817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd61d86f21e94ddcb13753609c13a2f7: Sending DeleteTablet for 3 replicas of tablet 83f9fc2b14de412db3caaf8914efaa5c
I20260925 10:07:03.998668 6201 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='slave'} at 127.0.0.1:52126
I20260925 10:07:03.998694 5940 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='slave'} at 127.0.0.1:52836
I20260925 10:07:03.998744 6071 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f9fc2b14de412db3caaf8914efaa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:03 UTC) from {username='slave'} at 127.0.0.1:40838
I20260925 10:07:03.998926 6305 tablet_replica.cc:330] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: stopping tablet replica
I20260925 10:07:03.999083 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5878
I20260925 10:07:03.999109 6305 raft_consensus.cc:2229] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:03.999219 6305 raft_consensus.cc:2258] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:03.999701 6305 ts_tablet_manager.cc:1905] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:04.002014 6305 ts_tablet_manager.cc:1918] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:04.002111 6305 log.cc:1196] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/83f9fc2b14de412db3caaf8914efaa5c
I20260925 10:07:04.002406 6305 ts_tablet_manager.cc:1939] T 83f9fc2b14de412db3caaf8914efaa5c P e8c5fee42e1f472db4a9e24bc99f5172: Deleting consensus metadata
I20260925 10:07:04.002934 5804 catalog_manager.cc:4841] TS e8c5fee42e1f472db4a9e24bc99f5172 (127.19.46.195:36051): tablet 83f9fc2b14de412db3caaf8914efaa5c (table test-table [id=64ec5880662f4de990f20b7e8a78feda]) successfully deleted
I20260925 10:07:04.002986 6307 tablet_replica.cc:330] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: stopping tablet replica
I20260925 10:07:04.003113 6307 raft_consensus.cc:2229] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:04.003769 6307 raft_consensus.cc:2258] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:04.004858 6307 ts_tablet_manager.cc:1905] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:04.007612 6307 ts_tablet_manager.cc:1918] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:04.007738 6307 log.cc:1196] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/83f9fc2b14de412db3caaf8914efaa5c
I20260925 10:07:04.008028 6307 ts_tablet_manager.cc:1939] T 83f9fc2b14de412db3caaf8914efaa5c P 6931fe9ef85f4892a0c0670a6edcd18b: Deleting consensus metadata
I20260925 10:07:04.008647 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6009
W20260925 10:07:04.008708 5805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:32775 (error 108)
I20260925 10:07:04.010033 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:04.009693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:32775 (local address 127.0.0.1:52850)
0925 10:07:04.009832 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:04.009964 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:32775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 10:07:04.010368 5805 catalog_manager.cc:4584] TS 920a41100f654ece9cd81a8e08a6991c (127.19.46.193:32775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83f9fc2b14de412db3caaf8914efaa5c: Network error: Client connection negotiation failed: client connection to 127.19.46.193:32775: connect: Connection refused (error 111)
I20260925 10:07:04.016887 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6140
I20260925 10:07:04.018000 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:04.016975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:37641 (local address 127.0.0.1:40842)
0925 10:07:04.017881 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:04.017932 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.194:37641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":866}
I20260925 10:07:04.023821 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 5787
2026-09-25T10:07:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (836 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:04 dist-test-slave-f9rl krb5kdc[6311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:04 dist-test-slave-f9rl krb5kdc[6311](info): set up 2 sockets
Sep 25 10:07:04 dist-test-slave-f9rl krb5kdc[6311](info): commencing operation
krb5kdc: starting...
W20260925 10:07:05.211539 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:05.360045 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37117
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:05.455549 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:05.455989 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:05.456048 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:05.457818 6324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:05.457933 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:05.457962 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:05.457998 6324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:05.458029 6324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:05.459815 6324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:05.460167 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:05.460433 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:05.467042 6330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.467036 6332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.467032 6329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:05.467056 6324 server_base.cc:1034] running on GCE node
I20260925 10:07:05.467772 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:05.468089 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:05.469293 6324 hybrid_clock.cc:648] HybridClock initialized: now 1790330825469273 us; error 53 us; skew 500 ppm
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:05.472352 6324 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:05.472970 6324 webserver.cc:460] Webserver started at http://127.19.46.254:44517/ using document root <none> and password file <none>
I20260925 10:07:05.473243 6324 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:05.473325 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:05.473547 6324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:05.475173 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4905507888af4b1d926f91dad7ac9600"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "4ff5d1f6f552b150037d83d160746441"
server_key_iv: "75b3a32a364d0ff5d18ca437ad276828"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.475791 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4905507888af4b1d926f91dad7ac9600"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "4ff5d1f6f552b150037d83d160746441"
server_key_iv: "75b3a32a364d0ff5d18ca437ad276828"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.478320 6324 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:07:05.479524 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.479966 6324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:05.480151 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4905507888af4b1d926f91dad7ac9600"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "4ff5d1f6f552b150037d83d160746441"
server_key_iv: "75b3a32a364d0ff5d18ca437ad276828"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.480278 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:05.515059 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:05.518931 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:05.519143 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:05.524720 6324 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:37117
I20260925 10:07:05.524875 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:37117 every 8 connection(s)
I20260925 10:07:05.525213 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:05.526592 6392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:05.528329 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6324
I20260925 10:07:05.528448 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:05.528641 19643 external_mini_cluster.cc:1428] Setting key 65dffbdcdf789b7a2957a9fb4a5e4e6b
I20260925 10:07:05.533777 6392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: Bootstrap starting.
I20260925 10:07:05.534893 6392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:05.535233 6392 log.cc:824] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:05.536499 6392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: No bootstrap required, opened a new log
I20260925 10:07:05.540290 6392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } }
I20260925 10:07:05.540500 6392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:05.540823 6392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4905507888af4b1d926f91dad7ac9600, State: Initialized, Role: FOLLOWER
I20260925 10:07:05.541067 6392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } }
I20260925 10:07:05.541193 6392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:05.541224 6392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:05.541307 6392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:05.542689 6392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } }
I20260925 10:07:05.542970 6392 leader_election.cc:302] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4905507888af4b1d926f91dad7ac9600; no voters:
I20260925 10:07:05.543262 6392 leader_election.cc:288] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:05.543493 6397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:05.543665 6392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:05.544166 6397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [term 1 LEADER]: Becoming Leader. State: Replica: 4905507888af4b1d926f91dad7ac9600, State: Running, Role: LEADER
I20260925 10:07:05.544317 6397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } }
I20260925 10:07:05.544833 6397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4905507888af4b1d926f91dad7ac9600. Latest consensus state: current_term: 1 leader_uuid: "4905507888af4b1d926f91dad7ac9600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } } }
I20260925 10:07:05.544924 6397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:05.545068 6398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4905507888af4b1d926f91dad7ac9600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4905507888af4b1d926f91dad7ac9600" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37117 } } }
I20260925 10:07:05.545138 6398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:05.545359 6406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:05.546168 6406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:05.547979 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:05.529450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.19.46.254:37117)
0925 10:07:05.529700 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:05.529711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:05.529751 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:07:05.531278 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:05.531290 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:05.532078 (+ 788us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:05.532492 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:05.533488 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.535190 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.536454 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.537269 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.538045 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:05.542528 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:05.542537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:05.542538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:05.542563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:05.545769 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:05.546407 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:05.546411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:05.546411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:05.546547 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:05.546850 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:05.546852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:05.546852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:05.547198 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:05.547390 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:05.547568 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:07:05.547742 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260925 10:07:05.548287 6406 catalog_manager.cc:1347] Generated new cluster ID: 619becd3ca224e7f80d1838a3dfa3cb5
I20260925 10:07:05.548329 6406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:05.567289 6406 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:05.568658 6406 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:05.588886 6406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: Generated new TSK 0
I20260925 10:07:05.589272 6406 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
Entry for principal kudu/127.19.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab.
I20260925 10:07:05.635656 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:05.731163 6418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:05.731400 6418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:05.731426 6418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:05.732684 6418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:05.732736 6418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:05.732765 6418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:07:05.734259 6418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.193.keytab
--principal=kudu/127.19.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:05.734484 6418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:05.734678 6418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:05.740514 6423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.740545 6426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.740571 6424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:05.741048 6418 server_base.cc:1034] running on GCE node
I20260925 10:07:05.741285 6418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:05.741657 6418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:05.742863 6418 hybrid_clock.cc:648] HybridClock initialized: now 1790330825742847 us; error 45 us; skew 500 ppm
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:05.746429 6418 init.cc:377] Logged in from keytab as kudu/127.19.46.193@KRBTEST.COM (short username kudu)
I20260925 10:07:05.747128 6418 webserver.cc:460] Webserver started at http://127.19.46.193:43635/ using document root <none> and password file <none>
I20260925 10:07:05.747411 6418 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:05.747501 6418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:05.747676 6418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:05.748942 6418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "b46b5821c351401fbbad5933d1350266"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "47657199467fded342ee768baed427b8"
server_key_iv: "7543789d6c43666806d0a8dfb92314c9"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.749356 6418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "b46b5821c351401fbbad5933d1350266"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "47657199467fded342ee768baed427b8"
server_key_iv: "7543789d6c43666806d0a8dfb92314c9"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.751067 6418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:05.752564 6433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.752935 6418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:05.753139 6418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "b46b5821c351401fbbad5933d1350266"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "47657199467fded342ee768baed427b8"
server_key_iv: "7543789d6c43666806d0a8dfb92314c9"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.753247 6418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:05.767125 6418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:05.769675 6418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:05.769848 6418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:05.770125 6418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:05.770444 6418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:05.770500 6418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.770543 6418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:05.770583 6418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.778160 6418 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:44895
I20260925 10:07:05.778513 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:44895 every 8 connection(s)
I20260925 10:07:05.778774 6418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:07:05.782981 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6418
I20260925 10:07:05.783340 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:05.783630 19643 external_mini_cluster.cc:1428] Setting key 6d4f5bb36c55f4f968c45ca184fe0d92
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.193@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:05.794087 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:05.780024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:42077 (local address 127.19.46.254:37117)
0925 10:07:05.780081 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:05.780083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:05.780261 (+ 178us) server_negotiation.cc:407] Connection header received
0925 10:07:05.780839 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:05.780840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:05.780850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:05.781048 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:05.782288 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.782973 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.785231 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.785594 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.785667 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:05.788860 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:05.788863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:05.788864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:05.788876 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:05.791346 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:05.792172 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:05.792175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:05.792175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:05.792265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:05.792807 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:05.792808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:05.792809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:05.792994 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:05.793083 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:05.793854 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 10:07:05.793969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.19.46.194@KRBTEST.COM; defaulting to no policy
I20260925 10:07:05.794461 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:05.779904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37117 (local address 127.19.46.193:42077)
0925 10:07:05.780102 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:05.780153 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:05.780732 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:05.781108 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:05.781113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:05.781894 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:05.782157 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:05.782163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.783249 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.783250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:05.784952 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:05.784958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.785704 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.785705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:05.785919 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:05.786838 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:05.786859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:05.788700 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:05.791668 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:05.791686 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:05.791690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:05.792048 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:05.792509 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:05.792511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:05.792511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:05.792713 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:05.793234 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:05.793245 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:05.793630 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 10:07:05.793817 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 10:07:05.794093 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":150,"threads_started":1}
I20260925 10:07:05.795316 6547 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37117
I20260925 10:07:05.795424 6547 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:05.795637 6547 heartbeater.cc:511] Master 127.19.46.254:37117 requested a full tablet report, sending...
I20260925 10:07:05.796279 6356 ts_manager.cc:194] Registered new tserver with Master: b46b5821c351401fbbad5933d1350266 (127.19.46.193:44895)
I20260925 10:07:05.796916 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.193@KRBTEST.COM'} at 127.19.46.193:42077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
Entry for principal kudu/127.19.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab.
I20260925 10:07:05.818166 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:05.927496 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:05.927814 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:05.927858 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:05.929265 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:05.929365 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:05.929423 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:07:05.931443 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.194.keytab
--principal=kudu/127.19.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:05.931811 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:05.932133 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:05.938028 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.938387 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:05.938465 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:05.939261 6552 server_base.cc:1034] running on GCE node
I20260925 10:07:05.939620 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:05.939927 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:05.941141 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790330825941154 us; error 97 us; skew 500 ppm
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:05.945845 6552 init.cc:377] Logged in from keytab as kudu/127.19.46.194@KRBTEST.COM (short username kudu)
I20260925 10:07:05.946545 6552 webserver.cc:460] Webserver started at http://127.19.46.194:38109/ using document root <none> and password file <none>
I20260925 10:07:05.946879 6552 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:05.946934 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:05.947079 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:05.948266 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "bb21641c6ef64ac99c40e2e386f10243"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "545f6094ff00f0bb5e884e43ff850a93"
server_key_iv: "8f92803a4569cd0cff0b91a6114a0478"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.948611 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "bb21641c6ef64ac99c40e2e386f10243"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "545f6094ff00f0bb5e884e43ff850a93"
server_key_iv: "8f92803a4569cd0cff0b91a6114a0478"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.950186 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 10:07:05.951153 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.951488 6552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:05.951678 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "bb21641c6ef64ac99c40e2e386f10243"
format_stamp: "Formatted at 2026-09-25 10:07:05 on dist-test-slave-f9rl"
server_key: "545f6094ff00f0bb5e884e43ff850a93"
server_key_iv: "8f92803a4569cd0cff0b91a6114a0478"
server_key_version: "encryptionkey@0"
I20260925 10:07:05.951769 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:05.978215 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:05.982677 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:05.983224 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:05.984112 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:05.984508 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:05.984613 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.984653 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:05.984668 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:05.993270 6552 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:42145
I20260925 10:07:05.993314 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:42145 every 8 connection(s)
I20260925 10:07:05.993888 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:05.995811 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6552
I20260925 10:07:05.995945 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:05.996168 19643 external_mini_cluster.cc:1428] Setting key 7e754abed52ada9174a26469d5af20b9
Sep 25 10:07:05 dist-test-slave-f9rl krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330825, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.194@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.195@KRBTEST.COM; defaulting to no policy
I20260925 10:07:06.007332 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:05.994836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:53885 (local address 127.19.46.254:37117)
0925 10:07:05.994938 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:05.994940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:05.995215 (+ 275us) server_negotiation.cc:407] Connection header received
0925 10:07:05.995964 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:05.995966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:05.995981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:05.996122 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:05.997334 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.998231 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.999454 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.999666 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.999757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.002724 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:06.002727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:06.002727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:06.002738 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:06.004797 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.005783 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.005785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.005785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.005860 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.006337 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.006338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.006338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.006529 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:06.006636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.007167 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.007247 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:07:06.007571 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:05.994825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37117 (local address 127.19.46.194:53885)
0925 10:07:05.995074 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:05.995122 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:05.995812 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:05.996216 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:05.996223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:05.997007 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:05.997225 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:05.997231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.998357 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.998358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:05.999332 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:05.999334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:05.999770 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:05.999771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:05.999920 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.000507 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:06.000520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:06.002600 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:06.004990 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:06.005009 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:06.005013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:06.005587 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:06.006016 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:06.006019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:06.006019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:06.006243 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:06.006659 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:06.006669 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:06.006961 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.007118 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.007283 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
I20260925 10:07:06.008811 6681 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37117
I20260925 10:07:06.008940 6681 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:06.009220 6681 heartbeater.cc:511] Master 127.19.46.254:37117 requested a full tablet report, sending...
I20260925 10:07:06.010040 6356 ts_manager.cc:194] Registered new tserver with Master: bb21641c6ef64ac99c40e2e386f10243 (127.19.46.194:42145)
I20260925 10:07:06.010457 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.194@KRBTEST.COM'} at 127.19.46.194:53885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
Entry for principal kudu/127.19.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab.
I20260925 10:07:06.032087 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:06.145597 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:06.145969 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:06.146023 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:06.148738 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:06.148881 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:06.148952 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:07:06.152227 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.195.keytab
--principal=kudu/127.19.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:06.152926 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:06.153388 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:06.160221 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.160221 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.160449 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:06.160563 6686 server_base.cc:1034] running on GCE node
I20260925 10:07:06.160809 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:06.161094 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:06.162297 6686 hybrid_clock.cc:648] HybridClock initialized: now 1790330826162282 us; error 32 us; skew 500 ppm
Sep 25 10:07:06 dist-test-slave-f9rl krb5kdc[6311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330826, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:06.166344 6686 init.cc:377] Logged in from keytab as kudu/127.19.46.195@KRBTEST.COM (short username kudu)
I20260925 10:07:06.166972 6686 webserver.cc:460] Webserver started at http://127.19.46.195:38471/ using document root <none> and password file <none>
I20260925 10:07:06.167248 6686 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:06.167423 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:06.167654 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:06.168965 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "eb6900510041436e89826cba3bd0d92d"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "8f42289d80be6afdb1f34f224f84ef38"
server_key_iv: "31e19c30afe506d175cffc596b65fb65"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.169385 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "eb6900510041436e89826cba3bd0d92d"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "8f42289d80be6afdb1f34f224f84ef38"
server_key_iv: "31e19c30afe506d175cffc596b65fb65"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.171232 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:07:06.172367 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.172974 6686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:06.173355 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "eb6900510041436e89826cba3bd0d92d"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "8f42289d80be6afdb1f34f224f84ef38"
server_key_iv: "31e19c30afe506d175cffc596b65fb65"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.173643 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:06.197139 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:06.199754 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:06.199940 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:06.200204 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:06.200567 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:06.200623 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.200668 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:06.200701 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.207854 6686 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:33029
I20260925 10:07:06.207897 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:33029 every 8 connection(s)
I20260925 10:07:06.208248 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:06.210721 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6686
I20260925 10:07:06.210860 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:06.211061 19643 external_mini_cluster.cc:1428] Setting key a56802b7aa9440d79bd9650865aec512
Sep 25 10:07:06 dist-test-slave-f9rl krb5kdc[6311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330826, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.195@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:06.220225 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.208975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:41603 (local address 127.19.46.254:37117)
0925 10:07:06.209079 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.209081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.209305 (+ 224us) server_negotiation.cc:407] Connection header received
0925 10:07:06.209984 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.209986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.209996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.210050 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:06.211186 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.212355 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.213689 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.213925 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.214015 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.216460 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:06.216462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:06.216462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:06.216470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:06.218307 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.218912 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.218914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.218915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.219032 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.219444 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.219446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.219446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.219608 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:06.219686 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.220061 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.220150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:07:06.220372 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.208945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37117 (local address 127.19.46.195:41603)
0925 10:07:06.209153 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.209194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.209885 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.210107 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.210111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.210722 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:06.211022 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.211030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.212616 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.212617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.213614 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.213618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.214032 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.214033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.214164 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.214744 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:06.214753 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:06.216336 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:06.218498 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:06.218507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:06.218509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:06.218766 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:06.219158 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:06.219159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:06.219160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:06.219286 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:06.219710 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:06.219716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:06.219860 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.219974 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.220121 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260925 10:07:06.221086 6815 heartbeater.cc:347] Connected to a master server at 127.19.46.254:37117
I20260925 10:07:06.221189 6815 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:06.221351 6815 heartbeater.cc:511] Master 127.19.46.254:37117 requested a full tablet report, sending...
I20260925 10:07:06.222097 6356 ts_manager.cc:194] Registered new tserver with Master: eb6900510041436e89826cba3bd0d92d (127.19.46.195:33029)
I20260925 10:07:06.222688 6356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.46.195@KRBTEST.COM'} at 127.19.46.195:41603
I20260925 10:07:06.223857 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:07:06.232537 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.225580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.19.46.254:37117)
0925 10:07:06.225648 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.225651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.225818 (+ 167us) server_negotiation.cc:407] Connection header received
0925 10:07:06.225857 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.225858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.225868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.225917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:06.226519 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.227069 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.228261 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.228453 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.228519 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.229914 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:06.229915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:06.229916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:06.229923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:06.231397 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.231741 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.231742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.231742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.231792 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:06.231957 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:06.231958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:06.231958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:06.232114 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:06.232192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.232360 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.232432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:07:06.234402 6356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:07:06.236013 6356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:07:06.244778 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.19.46.195:33029)
0925 10:07:06.240199 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.240203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.240213 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:06.240251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.240253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.240269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.240358 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:06.240635 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.241975 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.243397 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.244153 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.244258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.244329 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.244641 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.244685 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":84,"threads_started":1}
I20260925 10:07:06.244777 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:33029 (local address 127.0.0.1:53794)
0925 10:07:06.239874 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.239889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.240005 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.240370 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.240374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.240377 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:06.240533 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.240536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.242088 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.242089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.243288 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.243290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.244263 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.244265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.244403 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.244440 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.244521 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.244639 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":88,"threads_started":1}
I20260925 10:07:06.245162 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:42145 (local address 127.0.0.1:56192)
0925 10:07:06.240011 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.240022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.240069 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.240596 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.240596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.240599 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:06.240645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.240646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.242333 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.242334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.243437 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.243440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.244878 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.244879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.244955 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.244960 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.245012 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.245067 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260925 10:07:06.245576 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.19.46.194:42145)
0925 10:07:06.240346 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.240351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.240361 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:06.240399 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.240401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.240418 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.240569 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:06.240711 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.242161 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.243545 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.244701 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.244774 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.245386 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.245417 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.245484 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":157,"threads_started":1}
I20260925 10:07:06.245784 6749 tablet_service.cc:1467] Processing CreateTablet for tablet f74f6a97d88a4bcfa781745f3a48422b (DEFAULT_TABLE table=test-table [id=3118af9a8bc64e71b407db8ee00ca01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:06.246243 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74f6a97d88a4bcfa781745f3a48422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:06.246208 6615 tablet_service.cc:1467] Processing CreateTablet for tablet f74f6a97d88a4bcfa781745f3a48422b (DEFAULT_TABLE table=test-table [id=3118af9a8bc64e71b407db8ee00ca01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:06.246613 6615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74f6a97d88a4bcfa781745f3a48422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:06.249346 6832 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Bootstrap starting.
I20260925 10:07:06.249536 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44895 (local address 127.0.0.1:60374)
0925 10:07:06.239578 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.239619 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.239681 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.240459 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.240460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.240462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:06.240532 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.240534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.241627 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.241628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.243139 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.243142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.245124 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.245125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.245199 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.245203 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.249323 (+ 4120us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.249431 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":84,"threads_started":1}
I20260925 10:07:06.250056 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.239695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.19.46.193:44895)
0925 10:07:06.240184 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.240189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.240201 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:06.240248 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.240250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.240269 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.240359 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:06.240632 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.241519 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.243225 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.245027 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.249727 (+ 4700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.249829 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.249875 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.249957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":85,"threads_started":1}
I20260925 10:07:06.250244 6832 tablet_bootstrap.cc:654] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:06.250947 6832 log.cc:824] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:06.250934 6481 tablet_service.cc:1467] Processing CreateTablet for tablet f74f6a97d88a4bcfa781745f3a48422b (DEFAULT_TABLE table=test-table [id=3118af9a8bc64e71b407db8ee00ca01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:06.251359 6481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74f6a97d88a4bcfa781745f3a48422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:06.251439 6833 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Bootstrap starting.
I20260925 10:07:06.252128 6832 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: No bootstrap required, opened a new log
I20260925 10:07:06.252166 6833 tablet_bootstrap.cc:654] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:06.252197 6832 ts_tablet_manager.cc:1397] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:07:06.252760 6833 log.cc:824] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:06.253665 6833 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: No bootstrap required, opened a new log
I20260925 10:07:06.253737 6833 ts_tablet_manager.cc:1397] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:07:06.254462 6832 raft_consensus.cc:348] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.254673 6832 raft_consensus.cc:374] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:06.254716 6832 raft_consensus.cc:729] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6900510041436e89826cba3bd0d92d, State: Initialized, Role: FOLLOWER
I20260925 10:07:06.254944 6832 consensus_queue.cc:260] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.254963 6836 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Bootstrap starting.
I20260925 10:07:06.255231 6832 ts_tablet_manager.cc:1428] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:07:06.255380 6815 heartbeater.cc:503] Master 127.19.46.254:37117 was elected leader, sending a full tablet report...
I20260925 10:07:06.256042 6833 raft_consensus.cc:348] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.256251 6833 raft_consensus.cc:374] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:06.256331 6833 raft_consensus.cc:729] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb21641c6ef64ac99c40e2e386f10243, State: Initialized, Role: FOLLOWER
I20260925 10:07:06.256404 6836 tablet_bootstrap.cc:654] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:06.256456 6833 consensus_queue.cc:260] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.256861 6681 heartbeater.cc:503] Master 127.19.46.254:37117 was elected leader, sending a full tablet report...
I20260925 10:07:06.256757 6833 ts_tablet_manager.cc:1428] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:07:06.257151 6836 log.cc:824] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:06.258098 6836 tablet_bootstrap.cc:492] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: No bootstrap required, opened a new log
I20260925 10:07:06.258184 6836 ts_tablet_manager.cc:1397] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:07:06.259686 6836 raft_consensus.cc:348] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.259840 6836 raft_consensus.cc:374] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:06.259865 6836 raft_consensus.cc:729] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46b5821c351401fbbad5933d1350266, State: Initialized, Role: FOLLOWER
I20260925 10:07:06.260010 6836 consensus_queue.cc:260] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.260497 6836 ts_tablet_manager.cc:1428] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:07:06.260617 6547 heartbeater.cc:503] Master 127.19.46.254:37117 was elected leader, sending a full tablet report...
W20260925 10:07:06.280064 6548 tablet.cc:2369] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:06.430332 6840 raft_consensus.cc:482] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:06.430576 6840 raft_consensus.cc:504] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.431025 6840 leader_election.cc:288] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6900510041436e89826cba3bd0d92d (127.19.46.195:33029), bb21641c6ef64ac99c40e2e386f10243 (127.19.46.194:42145)
I20260925 10:07:06.435464 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.431213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:33029 (local address 127.19.46.193:38439)
0925 10:07:06.431370 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.431384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.431450 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.431649 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.431650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.431652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:06.431725 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.431727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.432583 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.432584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.433943 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.433946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.434884 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.434887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.435189 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.435200 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.435288 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.435346 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 10:07:06.436079 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.431417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:38439 (local address 127.19.46.195:33029)
0925 10:07:06.431466 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.431468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.431480 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:06.431514 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.431515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.431525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.431590 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:06.431803 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.432505 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.434097 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.434513 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.434821 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.434850 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.435919 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.436004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:07:06.436299 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.431370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:42145 (local address 127.19.46.193:44921)
0925 10:07:06.431762 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.431772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.431855 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.432377 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.432378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.432380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:06.432526 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.432529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.433494 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.433495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.434996 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.435003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.436021 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.436022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.436106 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.436110 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.436172 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.436223 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260925 10:07:06.436545 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.432084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:44921 (local address 127.19.46.194:42145)
0925 10:07:06.432220 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.432222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.432231 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:06.432262 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.432263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.432275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.432333 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:06.432657 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.433377 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.435210 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.435881 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.435968 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.435978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.436241 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.436358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 10:07:06.436885 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74f6a97d88a4bcfa781745f3a48422b" candidate_uuid: "b46b5821c351401fbbad5933d1350266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6900510041436e89826cba3bd0d92d" is_pre_election: true
I20260925 10:07:06.437069 6635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74f6a97d88a4bcfa781745f3a48422b" candidate_uuid: "b46b5821c351401fbbad5933d1350266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb21641c6ef64ac99c40e2e386f10243" is_pre_election: true
I20260925 10:07:06.437119 6769 raft_consensus.cc:2454] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46b5821c351401fbbad5933d1350266 in term 0.
I20260925 10:07:06.437216 6635 raft_consensus.cc:2454] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46b5821c351401fbbad5933d1350266 in term 0.
I20260925 10:07:06.437394 6435 leader_election.cc:302] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b46b5821c351401fbbad5933d1350266, eb6900510041436e89826cba3bd0d92d; no voters:
I20260925 10:07:06.437592 6840 raft_consensus.cc:2781] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:06.437683 6840 raft_consensus.cc:482] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:06.437723 6840 raft_consensus.cc:3037] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:06.438597 6840 raft_consensus.cc:504] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.438783 6840 leader_election.cc:288] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [CANDIDATE]: Term 1 election: Requested vote from peers eb6900510041436e89826cba3bd0d92d (127.19.46.195:33029), bb21641c6ef64ac99c40e2e386f10243 (127.19.46.194:42145)
I20260925 10:07:06.438946 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74f6a97d88a4bcfa781745f3a48422b" candidate_uuid: "b46b5821c351401fbbad5933d1350266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6900510041436e89826cba3bd0d92d"
I20260925 10:07:06.439044 6769 raft_consensus.cc:3037] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:06.439059 6635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74f6a97d88a4bcfa781745f3a48422b" candidate_uuid: "b46b5821c351401fbbad5933d1350266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb21641c6ef64ac99c40e2e386f10243"
I20260925 10:07:06.439136 6635 raft_consensus.cc:3037] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:06.440083 6635 raft_consensus.cc:2454] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46b5821c351401fbbad5933d1350266 in term 1.
I20260925 10:07:06.440083 6769 raft_consensus.cc:2454] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46b5821c351401fbbad5933d1350266 in term 1.
I20260925 10:07:06.440263 6435 leader_election.cc:302] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b46b5821c351401fbbad5933d1350266, eb6900510041436e89826cba3bd0d92d; no voters:
I20260925 10:07:06.440368 6840 raft_consensus.cc:2781] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:06.440609 6840 raft_consensus.cc:686] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 1 LEADER]: Becoming Leader. State: Replica: b46b5821c351401fbbad5933d1350266, State: Running, Role: LEADER
I20260925 10:07:06.440728 6840 consensus_queue.cc:237] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } }
I20260925 10:07:06.441810 6356 catalog_manager.cc:5495] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 reported cstate change: term changed from 0 to 1, leader changed from <none> to b46b5821c351401fbbad5933d1350266 (127.19.46.193). New cstate: current_term: 1 leader_uuid: "b46b5821c351401fbbad5933d1350266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46b5821c351401fbbad5933d1350266" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 44895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 } health_report { overall_health: UNKNOWN } } }
W20260925 10:07:06.459113 6816 tablet.cc:2369] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:06.476677 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.473843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.19.46.193:44895)
0925 10:07:06.473937 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.473939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.473947 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:07:06.473986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.473987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.474014 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.474113 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:06.474299 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.474831 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.475712 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.475928 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.476017 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.476228 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:06.476371 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:06.476486 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.476534 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.476559 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:07:06.479460 6635 raft_consensus.cc:1261] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 1 FOLLOWER]: Refusing update from remote peer b46b5821c351401fbbad5933d1350266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:06.479460 6769 raft_consensus.cc:1261] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 1 FOLLOWER]: Refusing update from remote peer b46b5821c351401fbbad5933d1350266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:06.479943 6843 consensus_queue.cc:1035] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb21641c6ef64ac99c40e2e386f10243" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 42145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:06.480106 6840 consensus_queue.cc:1035] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6900510041436e89826cba3bd0d92d" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:06.483104 6848 mvcc.cc:205] Tried to move back new op lower bound from 7333195065254912000 to 7333195065101905920. Current Snapshot: MvccSnapshot[applied={T|T < 7333195065254912000}]
I20260925 10:07:06.484552 6847 mvcc.cc:205] Tried to move back new op lower bound from 7333195065254912000 to 7333195065101905920. Current Snapshot: MvccSnapshot[applied={T|T < 7333195065254912000 or (T in {7333195065254912000})}]
I20260925 10:07:06.485780 6356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:06.485893 6356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:06.486738 6356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4905507888af4b1d926f91dad7ac9600: Sending DeleteTablet for 3 replicas of tablet f74f6a97d88a4bcfa781745f3a48422b
I20260925 10:07:06.487170 6615 tablet_service.cc:1514] Processing DeleteTablet for tablet f74f6a97d88a4bcfa781745f3a48422b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:06 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:56192
I20260925 10:07:06.487191 6749 tablet_service.cc:1514] Processing DeleteTablet for tablet f74f6a97d88a4bcfa781745f3a48422b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:06 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260925 10:07:06.487448 6856 tablet_replica.cc:330] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: stopping tablet replica
I20260925 10:07:06.487440 6481 tablet_service.cc:1514] Processing DeleteTablet for tablet f74f6a97d88a4bcfa781745f3a48422b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:06 UTC) from {username='kudu', principal='kudu/127.19.46.254@KRBTEST.COM'} at 127.0.0.1:60374
I20260925 10:07:06.487597 6856 raft_consensus.cc:2229] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:06.487675 6858 tablet_replica.cc:330] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266: stopping tablet replica
I20260925 10:07:06.487772 6856 raft_consensus.cc:2258] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:06.487830 6858 raft_consensus.cc:2229] T f74f6a97d88a4bcfa781745f3a48422b P b46b5821c351401fbbad5933d1350266 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:06.487879 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6418
I20260925 10:07:06.489403 6857 tablet_replica.cc:330] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: stopping tablet replica
I20260925 10:07:06.489560 6857 raft_consensus.cc:2229] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:06.489681 6857 raft_consensus.cc:2258] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:06.490062 6856 ts_tablet_manager.cc:1905] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:06.491906 6856 ts_tablet_manager.cc:1918] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:06.492000 6856 log.cc:1196] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/f74f6a97d88a4bcfa781745f3a48422b
I20260925 10:07:06.491999 6857 ts_tablet_manager.cc:1905] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:06.492443 6856 ts_tablet_manager.cc:1939] T f74f6a97d88a4bcfa781745f3a48422b P eb6900510041436e89826cba3bd0d92d: Deleting consensus metadata
I20260925 10:07:06.493050 6341 catalog_manager.cc:4841] TS eb6900510041436e89826cba3bd0d92d (127.19.46.195:33029): tablet f74f6a97d88a4bcfa781745f3a48422b (table test-table [id=3118af9a8bc64e71b407db8ee00ca01c]) successfully deleted
I20260925 10:07:06.494935 6857 ts_tablet_manager.cc:1918] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:06.495070 6857 log.cc:1196] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/f74f6a97d88a4bcfa781745f3a48422b
I20260925 10:07:06.495404 6857 ts_tablet_manager.cc:1939] T f74f6a97d88a4bcfa781745f3a48422b P bb21641c6ef64ac99c40e2e386f10243: Deleting consensus metadata
W20260925 10:07:06.495848 6342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.46.193:44895 (error 108)
I20260925 10:07:06.496165 6342 catalog_manager.cc:4841] TS bb21641c6ef64ac99c40e2e386f10243 (127.19.46.194:42145): tablet f74f6a97d88a4bcfa781745f3a48422b (table test-table [id=3118af9a8bc64e71b407db8ee00ca01c]) successfully deleted
W20260925 10:07:06.496567 6826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:06.496399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:44895 (local address 127.0.0.1:60388)
0925 10:07:06.496471 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.496516 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44895: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:07:06.496701 6342 catalog_manager.cc:4584] TS b46b5821c351401fbbad5933d1350266 (127.19.46.193:44895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f74f6a97d88a4bcfa781745f3a48422b: Network error: Client connection negotiation failed: client connection to 127.19.46.193:44895: connect: Connection reset by peer (error 104)
I20260925 10:07:06.496781 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6552
I20260925 10:07:06.502617 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6686
I20260925 10:07:06.508282 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6324
2026-09-25T10:07:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2484 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:06Z Disabled control of system clock
I20260925 10:07:06.540565 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42343
--webserver_interface=127.19.46.254
--webserver_port=0
--builtin_ntp_servers=127.19.46.212:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:06.638633 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:06.638922 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:06.638950 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:06.640611 6863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:06.640671 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:06.640684 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:06.640699 6863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:06.640713 6863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:06.642488 6863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:42343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:06.642778 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:06.643067 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:06.649811 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.649811 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.649909 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:06.650214 6863 server_base.cc:1034] running on GCE node
I20260925 10:07:06.650473 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:06.650756 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:06.651983 6863 hybrid_clock.cc:648] HybridClock initialized: now 1790330826651953 us; error 31 us; skew 500 ppm
I20260925 10:07:06.654268 6863 webserver.cc:460] Webserver started at http://127.19.46.254:40027/ using document root <none> and password file <none>
I20260925 10:07:06.654590 6863 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:06.654654 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:06.654781 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:06.655958 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "46e9e1d40d31455e9c286b97b62a9ce1"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "bd7217338a39a06a8a1a633b6769eb0d"
server_key_iv: "d26a5c57227a8a0393a399c4d060e7ea"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.656356 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "46e9e1d40d31455e9c286b97b62a9ce1"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "bd7217338a39a06a8a1a633b6769eb0d"
server_key_iv: "d26a5c57227a8a0393a399c4d060e7ea"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.657928 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:06.658913 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.659183 6863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:06.659307 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "46e9e1d40d31455e9c286b97b62a9ce1"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "bd7217338a39a06a8a1a633b6769eb0d"
server_key_iv: "d26a5c57227a8a0393a399c4d060e7ea"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.659394 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:06.671066 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:06.671432 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:06.671603 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:06.676398 6863 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:42343
I20260925 10:07:06.676447 6930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:42343 every 8 connection(s)
I20260925 10:07:06.676862 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:06.677614 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6863
I20260925 10:07:06.677740 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:06.677953 19643 external_mini_cluster.cc:1428] Setting key 97583d19a0138a40a03049114d43c127
I20260925 10:07:06.678426 6931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:06.681818 6931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: Bootstrap starting.
I20260925 10:07:06.682546 6931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:06.682850 6931 log.cc:824] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:06.683683 6931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: No bootstrap required, opened a new log
I20260925 10:07:06.684543 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.678627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.19.46.254:42343)
0925 10:07:06.678837 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.678847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.678884 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:07:06.679616 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.679626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.679707 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.679833 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:06.680126 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.681143 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.682315 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.682831 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.682961 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.683719 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:06.683724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:06.683725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:06.683944 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:06.683985 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:06.684000 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:06.684072 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.684200 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.684315 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 10:07:06.686436 6931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } }
I20260925 10:07:06.686646 6931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:06.686705 6931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9e1d40d31455e9c286b97b62a9ce1, State: Initialized, Role: FOLLOWER
I20260925 10:07:06.686872 6931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } }
I20260925 10:07:06.686962 6931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:06.687000 6931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:06.687038 6931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:06.687816 6931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } }
I20260925 10:07:06.687942 6931 leader_election.cc:302] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e9e1d40d31455e9c286b97b62a9ce1; no voters:
I20260925 10:07:06.688163 6931 leader_election.cc:288] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:06.688205 6936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:06.688326 6936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: 46e9e1d40d31455e9c286b97b62a9ce1, State: Running, Role: LEADER
I20260925 10:07:06.688417 6936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } }
I20260925 10:07:06.688467 6931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:06.688822 6938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e9e1d40d31455e9c286b97b62a9ce1. Latest consensus state: current_term: 1 leader_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } } }
I20260925 10:07:06.689270 6938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:06.689189 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9e1d40d31455e9c286b97b62a9ce1" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 42343 } } }
I20260925 10:07:06.689590 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:06.689774 6946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:06.690383 6946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:06.692178 6946 catalog_manager.cc:1347] Generated new cluster ID: b2fe4837cc3944daaebcb82085630cb5
I20260925 10:07:06.692250 6946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:06.717693 6946 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:06.718436 6946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:06.722270 6946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: Generated new TSK 0
I20260925 10:07:06.722489 6946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:06.729414 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.193:0
--local_ip_for_outbound_sockets=127.19.46.193
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--builtin_ntp_servers=127.19.46.212:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:07:06.824846 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:06.825074 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:06.825101 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:06.826639 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:06.826700 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:06.826730 6955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.193
I20260925 10:07:06.828261 6955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.46.193
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.193
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:06.828541 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:06.828881 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:06.836930 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.837066 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:06.836971 6955 server_base.cc:1034] running on GCE node
W20260925 10:07:06.836928 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:06.837888 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:06.838537 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:06.839775 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790330826839750 us; error 40 us; skew 500 ppm
I20260925 10:07:06.842300 6955 webserver.cc:460] Webserver started at http://127.19.46.193:34733/ using document root <none> and password file <none>
I20260925 10:07:06.842672 6955 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:06.842734 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:06.842896 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:06.844094 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data/instance:
uuid: "00018cf5a46b4b009b70d93200faea70"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "31de10648e53659a34b54e864b9cc901"
server_key_iv: "b9bec64a34816574fe3d9faa4f841065"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.844470 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance:
uuid: "00018cf5a46b4b009b70d93200faea70"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "31de10648e53659a34b54e864b9cc901"
server_key_iv: "b9bec64a34816574fe3d9faa4f841065"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.846801 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260925 10:07:06.847950 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.848299 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:06.848453 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal
uuid: "00018cf5a46b4b009b70d93200faea70"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "31de10648e53659a34b54e864b9cc901"
server_key_iv: "b9bec64a34816574fe3d9faa4f841065"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.848609 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:06.864673 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:06.865073 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:06.865242 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:06.865535 6955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:06.866086 6955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:06.866149 6955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.866187 6955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:06.866218 6955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:06.873271 6955 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.193:35721
I20260925 10:07:06.873317 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.193:35721 every 8 connection(s)
I20260925 10:07:06.873780 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/data/info.pb
I20260925 10:07:06.875787 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 6955
I20260925 10:07:06.875900 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:06.876330 19643 external_mini_cluster.cc:1428] Setting key 1bf43a4ea4794fb01e9f64ac61b6e32b
I20260925 10:07:06.877718 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.194:0
--local_ip_for_outbound_sockets=127.19.46.194
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--builtin_ntp_servers=127.19.46.212:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:07:06.880656 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.874813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42343 (local address 127.19.46.193:48379)
0925 10:07:06.875151 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:06.875281 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:06.876041 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:06.876353 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:06.876357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:06.876367 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:06.877003 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.877009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.877812 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.877813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.878686 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:06.878689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.879125 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.879127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:06.879270 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.879945 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:06.879946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:06.879987 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:06.880015 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:06.880285 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:06.880292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:06.880295 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:07:06.880370 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:07:06.880469 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
I20260925 10:07:06.880648 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:06.874899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.193:48379 (local address 127.19.46.254:42343)
0925 10:07:06.875090 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:06.875094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:06.875444 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:07:06.876174 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:06.876175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:06.876190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:06.876261 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:06.877114 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.877700 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.878792 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:06.878995 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:06.879101 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:06.880130 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:06.880132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:06.880132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:06.880138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:06.880158 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:06.880162 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:06.880224 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:06.880394 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:07:06.880442 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 10:07:06.881325 7084 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42343
I20260925 10:07:06.881484 7084 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:06.881747 7084 heartbeater.cc:511] Master 127.19.46.254:42343 requested a full tablet report, sending...
I20260925 10:07:06.882619 6894 ts_manager.cc:194] Registered new tserver with Master: 00018cf5a46b4b009b70d93200faea70 (127.19.46.193:35721)
I20260925 10:07:06.883293 6894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.193:48379
W20260925 10:07:06.981097 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:06.981356 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:06.981384 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:06.982832 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:06.982894 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:06.982924 7087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.194
I20260925 10:07:06.984534 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.46.194
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.194
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:06.984853 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:06.985208 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:06.991626 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.991628 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:06.991605 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:06.992002 7087 server_base.cc:1034] running on GCE node
I20260925 10:07:06.992254 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:06.992564 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:06.993822 7087 hybrid_clock.cc:648] HybridClock initialized: now 1790330826993760 us; error 74 us; skew 500 ppm
I20260925 10:07:06.995903 7087 webserver.cc:460] Webserver started at http://127.19.46.194:36207/ using document root <none> and password file <none>
I20260925 10:07:06.996261 7087 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:06.996349 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:06.996528 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:06.997865 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data/instance:
uuid: "e7f4c8e5903f473fa6445173f47b8011"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "dbe5d459b69effd3da5c7e574977b8f0"
server_key_iv: "935ad53df1eed2aeeb55ee20642d2e2d"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.998296 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance:
uuid: "e7f4c8e5903f473fa6445173f47b8011"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "dbe5d459b69effd3da5c7e574977b8f0"
server_key_iv: "935ad53df1eed2aeeb55ee20642d2e2d"
server_key_version: "encryptionkey@0"
I20260925 10:07:06.999987 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:07.001428 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.001865 7087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:07.002007 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal
uuid: "e7f4c8e5903f473fa6445173f47b8011"
format_stamp: "Formatted at 2026-09-25 10:07:06 on dist-test-slave-f9rl"
server_key: "dbe5d459b69effd3da5c7e574977b8f0"
server_key_iv: "935ad53df1eed2aeeb55ee20642d2e2d"
server_key_version: "encryptionkey@0"
I20260925 10:07:07.002133 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:07.013918 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:07.014680 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:07.014910 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:07.015479 7087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:07.016038 7087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:07.016276 7087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.016456 7087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:07.016505 7087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.023488 7087 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.194:45337
I20260925 10:07:07.023527 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.194:45337 every 8 connection(s)
I20260925 10:07:07.023909 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:07.024735 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7087
I20260925 10:07:07.024880 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:07.025141 19643 external_mini_cluster.cc:1428] Setting key f1cffe739cb4d5f9f076547d635d92da
I20260925 10:07:07.026612 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.46.195:0
--local_ip_for_outbound_sockets=127.19.46.195
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--builtin_ntp_servers=127.19.46.212:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:07:07.032284 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.024746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:41131 (local address 127.19.46.254:42343)
0925 10:07:07.024821 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.024824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.025016 (+ 192us) server_negotiation.cc:407] Connection header received
0925 10:07:07.025815 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.025817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.025830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.026025 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:07.027174 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.027965 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.029696 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.030036 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.030119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.031816 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:07.031818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:07.031819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:07.031826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:07.031850 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:07.031855 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:07.031941 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.032111 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.032150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:07:07.032582 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.024646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42343 (local address 127.19.46.194:41131)
0925 10:07:07.024859 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.024935 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.025661 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.026287 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.026293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.026307 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:07.027058 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.027065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.028103 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.028105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.029595 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.029598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.030144 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.030145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.030273 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.031564 (+ 1291us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:07.031565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:07.031618 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:07.031664 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:07.031989 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:07.031997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:07.032001 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.032097 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.032214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260925 10:07:07.033627 7216 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42343
I20260925 10:07:07.033761 7216 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:07.034003 7216 heartbeater.cc:511] Master 127.19.46.254:42343 requested a full tablet report, sending...
I20260925 10:07:07.035075 6894 ts_manager.cc:194] Registered new tserver with Master: e7f4c8e5903f473fa6445173f47b8011 (127.19.46.194:45337)
I20260925 10:07:07.035511 6894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.194:41131
W20260925 10:07:07.129817 7219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:07.130066 7219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:07.130091 7219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:07.131392 7219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:07.131451 7219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:07.131480 7219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.46.195
I20260925 10:07:07.133041 7219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.46.195
--webserver_port=0
--tserver_master_addrs=127.19.46.254:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.46.195
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:07.133332 7219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:07.133716 7219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:07.141559 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:07.141551 7225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:07.141659 7227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:07.141646 7219 server_base.cc:1034] running on GCE node
I20260925 10:07:07.142030 7219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:07.142321 7219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:07.143553 7219 hybrid_clock.cc:648] HybridClock initialized: now 1790330827143531 us; error 46 us; skew 500 ppm
I20260925 10:07:07.145167 7219 webserver.cc:460] Webserver started at http://127.19.46.195:39455/ using document root <none> and password file <none>
I20260925 10:07:07.145491 7219 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:07.145633 7219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:07.145785 7219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:07.147097 7219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data/instance:
uuid: "e07000ba1018494784b1682bc8962438"
format_stamp: "Formatted at 2026-09-25 10:07:07 on dist-test-slave-f9rl"
server_key: "8040c290f70ec861f4fe6f7b1f4c9a43"
server_key_iv: "8cbdc5beb11b9e2af0349396c514682d"
server_key_version: "encryptionkey@0"
I20260925 10:07:07.147538 7219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance:
uuid: "e07000ba1018494784b1682bc8962438"
format_stamp: "Formatted at 2026-09-25 10:07:07 on dist-test-slave-f9rl"
server_key: "8040c290f70ec861f4fe6f7b1f4c9a43"
server_key_iv: "8cbdc5beb11b9e2af0349396c514682d"
server_key_version: "encryptionkey@0"
I20260925 10:07:07.149729 7219 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:07:07.150944 7233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.151533 7219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:07.151695 7219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal
uuid: "e07000ba1018494784b1682bc8962438"
format_stamp: "Formatted at 2026-09-25 10:07:07 on dist-test-slave-f9rl"
server_key: "8040c290f70ec861f4fe6f7b1f4c9a43"
server_key_iv: "8cbdc5beb11b9e2af0349396c514682d"
server_key_version: "encryptionkey@0"
I20260925 10:07:07.151809 7219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:07.176955 7219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:07.177320 7219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:07.177541 7219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:07.177786 7219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:07.178139 7219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:07.178194 7219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.178236 7219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:07.178263 7219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:07.184845 7219 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.195:45227
I20260925 10:07:07.184881 7347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.195:45227 every 8 connection(s)
I20260925 10:07:07.185333 7219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:07.191411 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.186250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.195:36027 (local address 127.19.46.254:42343)
0925 10:07:07.186344 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.186346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.186485 (+ 139us) server_negotiation.cc:407] Connection header received
0925 10:07:07.187024 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.187025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.187039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.187078 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:07.187788 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.188348 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.189750 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.190046 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.190145 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.191120 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:07.191122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:07.191122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:07.191127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:07.191141 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:07.191145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:07.191203 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.191304 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.191329 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:07:07.191601 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.186165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:42343 (local address 127.19.46.195:36027)
0925 10:07:07.186357 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.186392 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.186892 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.187196 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.187200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.187208 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:07.187701 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.187705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.188462 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.188463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.189552 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.189556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.190184 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.190185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.190311 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.190956 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:07:07.190956 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:07.190987 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:07:07.191016 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:07.191219 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:07.191224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:07.191226 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.191295 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.191385 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260925 10:07:07.192134 7348 heartbeater.cc:347] Connected to a master server at 127.19.46.254:42343
I20260925 10:07:07.192234 7348 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:07.192386 7348 heartbeater.cc:511] Master 127.19.46.254:42343 requested a full tablet report, sending...
I20260925 10:07:07.192875 6894 ts_manager.cc:194] Registered new tserver with Master: e07000ba1018494784b1682bc8962438 (127.19.46.195:45227)
I20260925 10:07:07.193306 6894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.46.195:36027
I20260925 10:07:07.194051 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7219
I20260925 10:07:07.194145 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:07.194322 19643 external_mini_cluster.cc:1428] Setting key aa6ae8badd24e24bded445513566b069
I20260925 10:07:07.195237 19643 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:07:07.199748 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.196727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.19.46.254:42343)
0925 10:07:07.196793 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.196795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.196804 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:07.196863 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.196864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.196874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.196931 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:07.197093 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.197646 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.198409 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.198607 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.198685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.199481 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:07.199483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:07.199483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:07.199488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:07.199510 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:07.199514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:07.199576 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.199629 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.199654 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:07:07.201566 6894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:07:07.203333 6894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:07:07.210846 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.205999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:35721 (local address 127.0.0.1:43684)
0925 10:07:07.206241 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.206284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.206359 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.206798 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.206799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.206802 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:07.206864 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.206866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.208222 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.208224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.209730 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.209733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.210493 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.210494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.210581 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.210596 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.210698 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.210758 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":172,"threads_started":1}
I20260925 10:07:07.210858 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.206338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.19.46.193:35721)
0925 10:07:07.206625 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.206631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.206640 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:07.206682 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.206684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.206698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.206942 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:07.206999 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.208108 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.209832 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.210388 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.210480 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.210581 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.210718 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.210758 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":184,"threads_started":1}
I20260925 10:07:07.211757 7018 tablet_service.cc:1467] Processing CreateTablet for tablet 857b94acba3e41e6b121d143cab47b38 (DEFAULT_TABLE table=test-table [id=4728e648eb844bd09ba91463db4bac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:07.212258 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.206037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:45227 (local address 127.0.0.1:37814)
0925 10:07:07.206631 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.206643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.206738 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.208548 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.208549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.208551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:07.208606 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.208608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.209899 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.209899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.211109 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.211111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.212012 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.212013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.212075 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.212080 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.212132 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.212180 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":3,"thread_start_us":149,"threads_started":1}
I20260925 10:07:07.212260 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857b94acba3e41e6b121d143cab47b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:07.212273 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.206345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.19.46.195:45227)
0925 10:07:07.208291 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.208295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.208304 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:07.208337 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.208339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.208352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.208414 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:07.208719 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.209814 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.211196 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.211916 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.212011 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.212099 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.212160 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.212197 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":181,"threads_started":1}
I20260925 10:07:07.213490 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.206106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.19.46.194:45337)
0925 10:07:07.206502 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.206515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.206542 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:07:07.206587 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.206589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.206606 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.206694 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:07.206922 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.208109 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.211307 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.212237 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.213086 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.213212 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.213254 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.213307 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":111,"threads_started":1}
I20260925 10:07:07.213328 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.206005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.194:45337 (local address 127.0.0.1:46742)
0925 10:07:07.206291 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.206299 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.206359 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.206736 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.206740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.206742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:07.206829 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.206831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.209519 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.209520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.210831 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.210833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.213118 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.213119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.213185 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.213189 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.213239 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.213274 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":15,"thread_start_us":81,"threads_started":1}
I20260925 10:07:07.214257 7281 tablet_service.cc:1467] Processing CreateTablet for tablet 857b94acba3e41e6b121d143cab47b38 (DEFAULT_TABLE table=test-table [id=4728e648eb844bd09ba91463db4bac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:07.214650 7281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857b94acba3e41e6b121d143cab47b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:07.216223 7362 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Bootstrap starting.
I20260925 10:07:07.216682 7150 tablet_service.cc:1467] Processing CreateTablet for tablet 857b94acba3e41e6b121d143cab47b38 (DEFAULT_TABLE table=test-table [id=4728e648eb844bd09ba91463db4bac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:07:07.217034 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857b94acba3e41e6b121d143cab47b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:07.217053 7362 tablet_bootstrap.cc:654] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:07.217329 7363 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Bootstrap starting.
I20260925 10:07:07.217571 7362 log.cc:824] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:07.218007 7363 tablet_bootstrap.cc:654] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:07.218299 7363 log.cc:824] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:07.219007 7363 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: No bootstrap required, opened a new log
I20260925 10:07:07.219151 7363 ts_tablet_manager.cc:1397] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:07:07.220127 7366 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Bootstrap starting.
I20260925 10:07:07.220765 7366 tablet_bootstrap.cc:654] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:07.220932 7362 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: No bootstrap required, opened a new log
I20260925 10:07:07.221015 7362 ts_tablet_manager.cc:1397] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 10:07:07.221117 7366 log.cc:824] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:07.221576 7363 raft_consensus.cc:348] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.221796 7363 raft_consensus.cc:374] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:07.221843 7363 raft_consensus.cc:729] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07000ba1018494784b1682bc8962438, State: Initialized, Role: FOLLOWER
I20260925 10:07:07.221998 7363 consensus_queue.cc:260] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.222078 7366 tablet_bootstrap.cc:492] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: No bootstrap required, opened a new log
I20260925 10:07:07.222139 7366 ts_tablet_manager.cc:1397] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:07:07.222270 7363 ts_tablet_manager.cc:1428] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 10:07:07.222404 7348 heartbeater.cc:503] Master 127.19.46.254:42343 was elected leader, sending a full tablet report...
I20260925 10:07:07.223690 7362 raft_consensus.cc:348] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.223829 7362 raft_consensus.cc:374] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:07.223851 7362 raft_consensus.cc:729] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00018cf5a46b4b009b70d93200faea70, State: Initialized, Role: FOLLOWER
I20260925 10:07:07.223973 7362 consensus_queue.cc:260] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.224246 7362 ts_tablet_manager.cc:1428] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:07:07.224309 7084 heartbeater.cc:503] Master 127.19.46.254:42343 was elected leader, sending a full tablet report...
I20260925 10:07:07.224936 7366 raft_consensus.cc:348] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.225131 7366 raft_consensus.cc:374] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:07.225170 7366 raft_consensus.cc:729] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f4c8e5903f473fa6445173f47b8011, State: Initialized, Role: FOLLOWER
I20260925 10:07:07.225282 7366 consensus_queue.cc:260] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.225610 7366 ts_tablet_manager.cc:1428] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:07:07.225831 7216 heartbeater.cc:503] Master 127.19.46.254:42343 was elected leader, sending a full tablet report...
W20260925 10:07:07.274749 7217 tablet.cc:2369] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:07:07.374773 7085 tablet.cc:2369] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:07:07.435983 7349 tablet.cc:2369] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:07:07.558357 7370 raft_consensus.cc:482] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:07:07.558501 7370 raft_consensus.cc:504] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.558926 7370 leader_election.cc:288] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07000ba1018494784b1682bc8962438 (127.19.46.195:45227), 00018cf5a46b4b009b70d93200faea70 (127.19.46.193:35721)
I20260925 10:07:07.562942 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.559254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:58291 (local address 127.19.46.193:35721)
0925 10:07:07.559314 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.559316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.559494 (+ 178us) server_negotiation.cc:407] Connection header received
0925 10:07:07.559535 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.559536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.559552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.559632 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:07.559916 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.560709 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.561926 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.562484 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.562593 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.562602 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.562812 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.562866 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:07:07.562999 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.559243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.46.194:40223 (local address 127.19.46.195:45227)
0925 10:07:07.559303 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.559306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.559537 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:07:07.559571 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.559572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.559582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.559639 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:07:07.559913 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.560665 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.562075 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.562615 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.562688 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.562694 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.562879 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.562923 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:07:07.563093 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.559129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.195:45227 (local address 127.19.46.194:40223)
0925 10:07:07.559327 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.559341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.559428 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.559685 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.559687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.559689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:07.559799 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.559802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.560769 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.560770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.561991 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.561994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.562711 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.562712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.562791 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.562795 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.562866 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.563005 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":101,"thread_start_us":95,"threads_started":1}
I20260925 10:07:07.563292 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.559205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:35721 (local address 127.19.46.194:58291)
0925 10:07:07.559347 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.559356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:07.559428 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:07.559685 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:07.559687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:07.559690 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:07:07.559799 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.559802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.560790 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.560791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.561835 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:07.561838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.562608 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.562609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:07.562713 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.562719 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:07:07.562790 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:07:07.562842 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260925 10:07:07.563500 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857b94acba3e41e6b121d143cab47b38" candidate_uuid: "e7f4c8e5903f473fa6445173f47b8011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07000ba1018494784b1682bc8962438" is_pre_election: true
I20260925 10:07:07.563609 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857b94acba3e41e6b121d143cab47b38" candidate_uuid: "e7f4c8e5903f473fa6445173f47b8011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00018cf5a46b4b009b70d93200faea70" is_pre_election: true
I20260925 10:07:07.563695 7302 raft_consensus.cc:2454] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f4c8e5903f473fa6445173f47b8011 in term 0.
I20260925 10:07:07.563779 7038 raft_consensus.cc:2454] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f4c8e5903f473fa6445173f47b8011 in term 0.
I20260925 10:07:07.563922 7105 leader_election.cc:302] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e07000ba1018494784b1682bc8962438, e7f4c8e5903f473fa6445173f47b8011; no voters:
I20260925 10:07:07.564087 7370 raft_consensus.cc:2781] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:07:07.564150 7370 raft_consensus.cc:482] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:07:07.564168 7370 raft_consensus.cc:3037] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:07.564921 7370 raft_consensus.cc:504] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.565173 7370 leader_election.cc:288] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [CANDIDATE]: Term 1 election: Requested vote from peers e07000ba1018494784b1682bc8962438 (127.19.46.195:45227), 00018cf5a46b4b009b70d93200faea70 (127.19.46.193:35721)
I20260925 10:07:07.565359 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857b94acba3e41e6b121d143cab47b38" candidate_uuid: "e7f4c8e5903f473fa6445173f47b8011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00018cf5a46b4b009b70d93200faea70"
I20260925 10:07:07.565478 7038 raft_consensus.cc:3037] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:07.566288 7038 raft_consensus.cc:2454] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f4c8e5903f473fa6445173f47b8011 in term 1.
I20260925 10:07:07.566406 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857b94acba3e41e6b121d143cab47b38" candidate_uuid: "e7f4c8e5903f473fa6445173f47b8011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07000ba1018494784b1682bc8962438"
I20260925 10:07:07.566468 7302 raft_consensus.cc:3037] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:07.566636 7102 leader_election.cc:302] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00018cf5a46b4b009b70d93200faea70, e7f4c8e5903f473fa6445173f47b8011; no voters:
I20260925 10:07:07.566737 7370 raft_consensus.cc:2781] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:07.566907 7370 raft_consensus.cc:686] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 1 LEADER]: Becoming Leader. State: Replica: e7f4c8e5903f473fa6445173f47b8011, State: Running, Role: LEADER
I20260925 10:07:07.567148 7370 consensus_queue.cc:237] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } }
I20260925 10:07:07.567461 7302 raft_consensus.cc:2454] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f4c8e5903f473fa6445173f47b8011 in term 1.
I20260925 10:07:07.568296 6894 catalog_manager.cc:5495] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f4c8e5903f473fa6445173f47b8011 (127.19.46.194). New cstate: current_term: 1 leader_uuid: "e7f4c8e5903f473fa6445173f47b8011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f4c8e5903f473fa6445173f47b8011" member_type: VOTER last_known_addr { host: "127.19.46.194" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 } health_report { overall_health: UNKNOWN } } }
I20260925 10:07:07.655695 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.651871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.19.46.194:45337)
0925 10:07:07.651981 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:07.651983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:07.652019 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:07:07.652074 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:07.652075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:07.652097 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:07.652158 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:07.652326 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.652994 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.654352 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:07.654619 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:07.654721 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:07.655053 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:07.655180 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:07.655268 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:07.655578 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 10:07:07.655612 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:07:07.658135 7038 raft_consensus.cc:1261] T 857b94acba3e41e6b121d143cab47b38 P 00018cf5a46b4b009b70d93200faea70 [term 1 FOLLOWER]: Refusing update from remote peer e7f4c8e5903f473fa6445173f47b8011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:07.658135 7302 raft_consensus.cc:1261] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 1 FOLLOWER]: Refusing update from remote peer e7f4c8e5903f473fa6445173f47b8011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:07.658625 7373 consensus_queue.cc:1035] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07000ba1018494784b1682bc8962438" member_type: VOTER last_known_addr { host: "127.19.46.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:07.658803 7370 consensus_queue.cc:1035] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00018cf5a46b4b009b70d93200faea70" member_type: VOTER last_known_addr { host: "127.19.46.193" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:07.661851 7377 mvcc.cc:205] Tried to move back new op lower bound from 7333195070084104192 to 7333195069715841024. Current Snapshot: MvccSnapshot[applied={T|T < 7333195070084104192}]
I20260925 10:07:07.663559 6893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:07.663689 6893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:07:07.664006 7378 mvcc.cc:205] Tried to move back new op lower bound from 7333195070084104192 to 7333195069715841024. Current Snapshot: MvccSnapshot[applied={T|T < 7333195070084104192 or (T in {7333195070084104192})}]
I20260925 10:07:07.664510 6893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46e9e1d40d31455e9c286b97b62a9ce1: Sending DeleteTablet for 3 replicas of tablet 857b94acba3e41e6b121d143cab47b38
I20260925 10:07:07.664860 7150 tablet_service.cc:1514] Processing DeleteTablet for tablet 857b94acba3e41e6b121d143cab47b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:07 UTC) from {username='slave'} at 127.0.0.1:46742
I20260925 10:07:07.665083 7281 tablet_service.cc:1514] Processing DeleteTablet for tablet 857b94acba3e41e6b121d143cab47b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:07:07 UTC) from {username='slave'} at 127.0.0.1:37814
I20260925 10:07:07.665196 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6955
I20260925 10:07:07.665943 7386 tablet_replica.cc:330] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: stopping tablet replica
I20260925 10:07:07.666061 7386 raft_consensus.cc:2229] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:07:07.666189 7386 raft_consensus.cc:2258] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:07.666675 7386 ts_tablet_manager.cc:1905] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:07:07.668738 7386 ts_tablet_manager.cc:1918] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:07.668819 7386 log.cc:1196] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-1/wal/wals/857b94acba3e41e6b121d143cab47b38
I20260925 10:07:07.669071 7386 ts_tablet_manager.cc:1939] T 857b94acba3e41e6b121d143cab47b38 P e7f4c8e5903f473fa6445173f47b8011: Deleting consensus metadata
I20260925 10:07:07.669735 6880 catalog_manager.cc:4841] TS e7f4c8e5903f473fa6445173f47b8011 (127.19.46.194:45337): tablet 857b94acba3e41e6b121d143cab47b38 (table test-table [id=4728e648eb844bd09ba91463db4bac5f]) successfully deleted
I20260925 10:07:07.669891 7387 tablet_replica.cc:330] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: stopping tablet replica
I20260925 10:07:07.670039 7387 raft_consensus.cc:2229] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:07:07.670176 7387 raft_consensus.cc:2258] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:07:07.670715 7387 ts_tablet_manager.cc:1905] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 10:07:07.671888 6878 connection.cc:535] client connection to 127.19.46.193:35721 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 10:07:07.671991 6878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:07:07.672117 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7087
I20260925 10:07:07.673508 7387 ts_tablet_manager.cc:1918] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:07:07.673599 7387 log.cc:1196] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Deleting WAL directory at /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330556283750-19643-0/minicluster-data/ts-2/wal/wals/857b94acba3e41e6b121d143cab47b38
I20260925 10:07:07.673764 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:07.673595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.193:35721 (local address 127.0.0.1:43698)
0925 10:07:07.673679 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:07.673720 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.193:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:07:07.673899 7387 ts_tablet_manager.cc:1939] T 857b94acba3e41e6b121d143cab47b38 P e07000ba1018494784b1682bc8962438: Deleting consensus metadata
W20260925 10:07:07.673964 6878 catalog_manager.cc:4584] TS 00018cf5a46b4b009b70d93200faea70 (127.19.46.193:35721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 857b94acba3e41e6b121d143cab47b38: Network error: Client connection negotiation failed: client connection to 127.19.46.193:35721: connect: Connection refused (error 111)
I20260925 10:07:07.674383 6881 catalog_manager.cc:4841] TS e07000ba1018494784b1682bc8962438 (127.19.46.195:45227): tablet 857b94acba3e41e6b121d143cab47b38 (table test-table [id=4728e648eb844bd09ba91463db4bac5f]) successfully deleted
I20260925 10:07:07.678143 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7219
I20260925 10:07:07.683777 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 6863
2026-09-25T10:07:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1172 ms)
[----------] 6 tests from EncryptionPolicyTest (9752 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:07 dist-test-slave-f9rl krb5kdc[7391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:07 dist-test-slave-f9rl krb5kdc[7391](info): set up 2 sockets
Sep 25 10:07:07 dist-test-slave-f9rl krb5kdc[7391](info): commencing operation
krb5kdc: starting...
W20260925 10:07:08.819739 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:08 dist-test-slave-f9rl krb5kdc[7391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:08.978801 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:09.079041 7404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:09.079375 7404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:09.079409 7404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:09.081374 7404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:09.081481 7404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:09.081501 7404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:09.081516 7404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:09.081529 7404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:09.083552 7404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:09.083901 7404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:09.084218 7404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:09.091903 7412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:09.091921 7409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:09.091921 7410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:09.092916 7404 server_base.cc:1034] running on GCE node
I20260925 10:07:09.093410 7404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:09.093902 7404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:09.095168 7404 hybrid_clock.cc:648] HybridClock initialized: now 1790330829095115 us; error 46 us; skew 500 ppm
Sep 25 10:07:09 dist-test-slave-f9rl krb5kdc[7391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:09.107717 7404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:09.108273 7404 webserver.cc:460] Webserver started at http://127.0.0.1:46323/ using document root <none> and password file <none>
I20260925 10:07:09.108543 7404 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:09.108595 7404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:09.108749 7404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:09.110080 7404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "58196491bb5b47faa2cdb2ff436cc620"
format_stamp: "Formatted at 2026-09-25 10:07:09 on dist-test-slave-f9rl"
server_key: "ea413667875ca57d1042613b5518c371"
server_key_iv: "64567745d0fcdae1ed802e49232a03ed"
server_key_version: "encryptionkey@0"
I20260925 10:07:09.110504 7404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "58196491bb5b47faa2cdb2ff436cc620"
format_stamp: "Formatted at 2026-09-25 10:07:09 on dist-test-slave-f9rl"
server_key: "ea413667875ca57d1042613b5518c371"
server_key_iv: "64567745d0fcdae1ed802e49232a03ed"
server_key_version: "encryptionkey@0"
I20260925 10:07:09.112098 7404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:09.113065 7419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:09.113377 7404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:09.113561 7404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "58196491bb5b47faa2cdb2ff436cc620"
format_stamp: "Formatted at 2026-09-25 10:07:09 on dist-test-slave-f9rl"
server_key: "ea413667875ca57d1042613b5518c371"
server_key_iv: "64567745d0fcdae1ed802e49232a03ed"
server_key_version: "encryptionkey@0"
I20260925 10:07:09.113662 7404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:09.126046 7404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:09.129213 7404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:09.129422 7404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:09.133970 7404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40227
I20260925 10:07:09.134193 7472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40227 every 8 connection(s)
I20260925 10:07:09.134654 7404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:09.135663 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7404
I20260925 10:07:09.135797 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:09.135839 7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:09.136015 19643 external_mini_cluster.cc:1428] Setting key c06b1c4dad768f573a684b117f32e95b
I20260925 10:07:09.139978 7473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620: Bootstrap starting.
I20260925 10:07:09.140873 7473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:09.141242 7473 log.cc:824] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:09.142205 7473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620: No bootstrap required, opened a new log
Sep 25 10:07:09 dist-test-slave-f9rl krb5kdc[7391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:07:09.145177 7473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260925 10:07:09.145393 7473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:09.145424 7473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58196491bb5b47faa2cdb2ff436cc620, State: Initialized, Role: FOLLOWER
I20260925 10:07:09.145653 7473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260925 10:07:09.145750 7473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:09.145789 7473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:09.145850 7473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:09.146783 7473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260925 10:07:09.146901 7473 leader_election.cc:302] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58196491bb5b47faa2cdb2ff436cc620; no voters:
I20260925 10:07:09.147089 7473 leader_election.cc:288] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:09.147153 7478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:09.147320 7478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [term 1 LEADER]: Becoming Leader. State: Replica: 58196491bb5b47faa2cdb2ff436cc620, State: Running, Role: LEADER
I20260925 10:07:09.147410 7473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:09.147423 7478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260925 10:07:09.147859 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58196491bb5b47faa2cdb2ff436cc620. Latest consensus state: current_term: 1 leader_uuid: "58196491bb5b47faa2cdb2ff436cc620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260925 10:07:09.147962 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:09.147864 7479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58196491bb5b47faa2cdb2ff436cc620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58196491bb5b47faa2cdb2ff436cc620" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260925 10:07:09.148226 7479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:09.148239 7484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:09.149637 7484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:09.150041 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:09.136673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41211 (local address 127.0.0.1:40227)
0925 10:07:09.136911 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:09.136922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:09.136961 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:07:09.137698 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:09.137707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:09.137764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:09.137919 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:09.138867 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:09.139934 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:09.141218 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:09.141862 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:09.141957 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:09.144972 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:09.144978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:09.144979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:09.144995 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:09.147427 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:09.148197 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:09.148201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:09.148201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:09.148333 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:09.148619 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:09.148620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:09.148620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:09.149016 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:09.149578 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:09.149672 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:07:09.149790 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":159,"threads_started":1}
I20260925 10:07:09.151081 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:09.151443 7484 catalog_manager.cc:1347] Generated new cluster ID: fc5ad2e238cb43e695aff7cb76501eb7
I20260925 10:07:09.151492 7484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:09.161711 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:09.152629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58913 (local address 127.0.0.1:40227)
0925 10:07:09.152666 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:09.152668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:09.152891 (+ 223us) server_negotiation.cc:407] Connection header received
0925 10:07:09.152922 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:09.152924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:09.152937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:09.153009 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:09.153763 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:09.154675 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:09.155722 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:09.156121 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:09.156207 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:09.158039 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:09.158041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:09.158041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:09.158051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:09.159957 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:09.160464 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:09.160466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:09.160466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:09.160538 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:09.160795 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:09.160796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:09.160796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:09.160965 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:09.161270 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:09.161339 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:07:09.161417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:07:09.169284 7484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:09.169958 7484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:09.187649 7484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58196491bb5b47faa2cdb2ff436cc620: Generated new TSK 0
I20260925 10:07:09.187943 7484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:09.193198 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7404
W20260925 10:07:09.198165 7389 connection.cc:535] client connection to 127.0.0.1:40227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40227): Connection reset by peer (error 104)
2026-09-25T10:07:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:09 dist-test-slave-f9rl krb5kdc[7504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:09 dist-test-slave-f9rl krb5kdc[7504](info): set up 2 sockets
Sep 25 10:07:09 dist-test-slave-f9rl krb5kdc[7504](info): commencing operation
krb5kdc: starting...
W20260925 10:07:10.351275 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:10 dist-test-slave-f9rl krb5kdc[7504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:10.494299 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:10.590744 7517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:10.591035 7517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:10.591081 7517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:10.592458 7517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:10.592545 7517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:10.592572 7517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:10.592595 7517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:10.592624 7517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:10.594376 7517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:10.594720 7517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:10.594975 7517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:10.600757 7523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:10.600755 7525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:10.600976 7522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:10.601049 7517 server_base.cc:1034] running on GCE node
I20260925 10:07:10.601619 7517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:10.602279 7517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:10.603672 7517 hybrid_clock.cc:648] HybridClock initialized: now 1790330830603649 us; error 45 us; skew 500 ppm
Sep 25 10:07:10 dist-test-slave-f9rl krb5kdc[7504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:10.606935 7517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:10.607461 7517 webserver.cc:460] Webserver started at http://127.0.0.1:38217/ using document root <none> and password file <none>
I20260925 10:07:10.607733 7517 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:10.607806 7517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:10.607969 7517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:10.609100 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "0c4deb92e1ba40bdbce5fb87b63a5373"
format_stamp: "Formatted at 2026-09-25 10:07:10 on dist-test-slave-f9rl"
server_key: "e745a7b5f74e35ead65913e31ed61caa"
server_key_iv: "bcbd26eb2799f4dd22e75b53ca1ed139"
server_key_version: "encryptionkey@0"
I20260925 10:07:10.609571 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "0c4deb92e1ba40bdbce5fb87b63a5373"
format_stamp: "Formatted at 2026-09-25 10:07:10 on dist-test-slave-f9rl"
server_key: "e745a7b5f74e35ead65913e31ed61caa"
server_key_iv: "bcbd26eb2799f4dd22e75b53ca1ed139"
server_key_version: "encryptionkey@0"
I20260925 10:07:10.611402 7517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:07:10.612824 7532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:10.613324 7517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:10.613507 7517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "0c4deb92e1ba40bdbce5fb87b63a5373"
format_stamp: "Formatted at 2026-09-25 10:07:10 on dist-test-slave-f9rl"
server_key: "e745a7b5f74e35ead65913e31ed61caa"
server_key_iv: "bcbd26eb2799f4dd22e75b53ca1ed139"
server_key_version: "encryptionkey@0"
I20260925 10:07:10.613626 7517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:10.623211 7517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:10.626268 7517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:10.626482 7517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:10.631457 7517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43267
I20260925 10:07:10.631523 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43267 every 8 connection(s)
I20260925 10:07:10.631930 7517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:10.633224 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:10.636443 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373: Bootstrap starting.
I20260925 10:07:10.637720 7585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:10.638113 7585 log.cc:824] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:10.639194 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373: No bootstrap required, opened a new log
I20260925 10:07:10.641099 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7517
I20260925 10:07:10.641204 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:10.641418 19643 external_mini_cluster.cc:1428] Setting key cd6f8d9fdd641fc0fc7339c934fc3680
I20260925 10:07:10.641566 7585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 10:07:10.641826 7585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:10.641875 7585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4deb92e1ba40bdbce5fb87b63a5373, State: Initialized, Role: FOLLOWER
I20260925 10:07:10.642112 7585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 10:07:10.642249 7585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:10.642297 7585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:10.642350 7585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:10.643725 7585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 10:07:10.643908 7585 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c4deb92e1ba40bdbce5fb87b63a5373; no voters:
I20260925 10:07:10.644243 7585 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:10.644270 7590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:10.644440 7590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4deb92e1ba40bdbce5fb87b63a5373, State: Running, Role: LEADER
I20260925 10:07:10.644550 7590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 10:07:10.644622 7585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:10.644924 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4deb92e1ba40bdbce5fb87b63a5373. Latest consensus state: current_term: 1 leader_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } } }
I20260925 10:07:10.644920 7592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4deb92e1ba40bdbce5fb87b63a5373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } } }
I20260925 10:07:10.645054 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:10.645066 7592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:10.645304 7597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:10.645925 7597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:10.647845 7597 catalog_manager.cc:1347] Generated new cluster ID: f25b0d3ee2134dc8a0e1ce76f70ed107
I20260925 10:07:10.647897 7597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:10.648236 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:10.642163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38545 (local address 127.0.0.1:43267)
0925 10:07:10.642444 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:10.642456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:10.642495 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:07:10.642624 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:10.642630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:10.642679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:10.642835 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:10.643085 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:10.644500 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:10.645718 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:10.646416 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:10.646908 (+ 492us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:10.647547 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:10.647550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:10.647551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:10.647717 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:10.647780 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:10.647798 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:10.647905 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:10.647992 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:07:10.648022 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":115,"threads_started":1}
I20260925 10:07:10.649188 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:10.654510 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:10.651009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50333 (local address 127.0.0.1:43267)
0925 10:07:10.651054 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:10.651057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:10.651352 (+ 295us) server_negotiation.cc:407] Connection header received
0925 10:07:10.651566 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:10.651567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:10.651580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:10.651632 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:10.651794 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:10.652353 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:10.653096 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:10.653338 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:10.653420 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:10.654212 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:10.654214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:10.654214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:10.654219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:10.654235 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:10.654240 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:10.654297 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:10.654382 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:07:10.654395 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:07:10.709374 7597 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:10.710191 7597 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:10.715660 7597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c4deb92e1ba40bdbce5fb87b63a5373: Generated new TSK 0
I20260925 10:07:10.715909 7597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:10.724644 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7517
W20260925 10:07:10.729395 7502 connection.cc:535] client connection to 127.0.0.1:43267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43267): Connection reset by peer (error 104)
2026-09-25T10:07:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:10 dist-test-slave-f9rl krb5kdc[7617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:10 dist-test-slave-f9rl krb5kdc[7617](info): set up 2 sockets
Sep 25 10:07:10 dist-test-slave-f9rl krb5kdc[7617](info): commencing operation
krb5kdc: starting...
W20260925 10:07:11.839449 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:11 dist-test-slave-f9rl krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:11.990463 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:12.082456 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:12.082784 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:12.082839 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:12.084578 7630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:12.084694 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:12.084723 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:12.084746 7630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:12.084775 7630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:12.086941 7630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:12.087318 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:12.087591 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:12.093369 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:12.093380 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:12.093400 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:12.093458 7630 server_base.cc:1034] running on GCE node
I20260925 10:07:12.093926 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:12.094285 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:12.095620 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790330832095580 us; error 59 us; skew 500 ppm
Sep 25 10:07:12 dist-test-slave-f9rl krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:12.098806 7630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:12.099411 7630 webserver.cc:460] Webserver started at http://127.0.0.1:46845/ using document root <none> and password file <none>
I20260925 10:07:12.099689 7630 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:12.099745 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:12.099892 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:12.101133 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "4154684f0878489aa1b129869581acf8"
format_stamp: "Formatted at 2026-09-25 10:07:12 on dist-test-slave-f9rl"
server_key: "a204af1b9b34d8a5887cc0628ccab7bd"
server_key_iv: "eee7b4a753cb6b382545de2ec4fe2961"
server_key_version: "encryptionkey@0"
I20260925 10:07:12.101760 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "4154684f0878489aa1b129869581acf8"
format_stamp: "Formatted at 2026-09-25 10:07:12 on dist-test-slave-f9rl"
server_key: "a204af1b9b34d8a5887cc0628ccab7bd"
server_key_iv: "eee7b4a753cb6b382545de2ec4fe2961"
server_key_version: "encryptionkey@0"
I20260925 10:07:12.103514 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:07:12.104521 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:12.104915 7630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:12.105057 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "4154684f0878489aa1b129869581acf8"
format_stamp: "Formatted at 2026-09-25 10:07:12 on dist-test-slave-f9rl"
server_key: "a204af1b9b34d8a5887cc0628ccab7bd"
server_key_iv: "eee7b4a753cb6b382545de2ec4fe2961"
server_key_version: "encryptionkey@0"
I20260925 10:07:12.105160 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:12.109772 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:12.112494 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:12.112694 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:12.117869 7630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35455
I20260925 10:07:12.117903 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35455 every 8 connection(s)
I20260925 10:07:12.118709 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:12.119784 7698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:12.122468 7698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8: Bootstrap starting.
I20260925 10:07:12.123172 7698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:12.123517 7698 log.cc:824] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:12.124373 7698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8: No bootstrap required, opened a new log
I20260925 10:07:12.126258 7698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260925 10:07:12.126441 7698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:12.126492 7698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154684f0878489aa1b129869581acf8, State: Initialized, Role: FOLLOWER
I20260925 10:07:12.126708 7698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260925 10:07:12.126816 7698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:12.126863 7698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:12.126796 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7630
I20260925 10:07:12.126911 7698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:12.126989 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:12.127200 19643 external_mini_cluster.cc:1428] Setting key 882e8531b11ef28fa256ea48a6e09d97
I20260925 10:07:12.127656 7698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260925 10:07:12.127775 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4154684f0878489aa1b129869581acf8; no voters:
I20260925 10:07:12.127940 7698 leader_election.cc:288] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:12.128027 7703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:12.128211 7698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:12.128384 7703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [term 1 LEADER]: Becoming Leader. State: Replica: 4154684f0878489aa1b129869581acf8, State: Running, Role: LEADER
I20260925 10:07:12.128522 7703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260925 10:07:12.129206 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4154684f0878489aa1b129869581acf8. Latest consensus state: current_term: 1 leader_uuid: "4154684f0878489aa1b129869581acf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } } }
I20260925 10:07:12.129370 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:12.129417 7704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4154684f0878489aa1b129869581acf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4154684f0878489aa1b129869581acf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } } }
I20260925 10:07:12.129557 7704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:12.129921 7716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:12.130489 7716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:12.131007 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:12.127737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52951 (local address 127.0.0.1:35455)
0925 10:07:12.127964 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:12.127974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:12.128030 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:07:12.128165 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:12.128172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:12.128204 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:12.128664 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:12.130192 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:12.130195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:12.130195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:12.130198 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:12.130221 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:12.130232 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:12.130699 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:12.130785 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:07:12.130835 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260925 10:07:12.132066 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:12.132359 7716 catalog_manager.cc:1347] Generated new cluster ID: aa6f873426d34543844add10e54627a1
I20260925 10:07:12.132428 7716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:12.138983 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:12.133700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59053 (local address 127.0.0.1:35455)
0925 10:07:12.133739 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:12.133741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:12.137734 (+ 3993us) server_negotiation.cc:407] Connection header received
0925 10:07:12.137819 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:12.137821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:12.137830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:12.137913 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:12.138703 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:12.138704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:12.138705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:12.138710 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:12.138729 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:12.138734 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:12.138795 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:12.138884 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:07:12.138899 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:07:12.145223 7716 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:12.145524 7716 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:12.152408 7716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4154684f0878489aa1b129869581acf8: Generated new TSK 0
I20260925 10:07:12.152711 7716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:12.163081 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7630
2026-09-25T10:07:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:12 dist-test-slave-f9rl krb5kdc[7729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:12 dist-test-slave-f9rl krb5kdc[7729](info): set up 2 sockets
Sep 25 10:07:12 dist-test-slave-f9rl krb5kdc[7729](info): commencing operation
krb5kdc: starting...
W20260925 10:07:13.332818 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:13.495906 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:13.613035 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:13.613358 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:13.613408 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:13.615126 7742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:13.615252 7742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:13.615284 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:13.615331 7742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:13.615366 7742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:13.617328 7742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:13.617713 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:13.617998 7742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:13.625643 7750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:13.626104 7742 server_base.cc:1034] running on GCE node
W20260925 10:07:13.625643 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:13.626029 7748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:13.627826 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:13.628371 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:13.629590 7742 hybrid_clock.cc:648] HybridClock initialized: now 1790330833629544 us; error 71 us; skew 500 ppm
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:13.633097 7742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:13.633823 7742 webserver.cc:460] Webserver started at http://127.0.0.1:44011/ using document root <none> and password file <none>
I20260925 10:07:13.634102 7742 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:13.634184 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:13.634359 7742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:13.635684 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "1cec78b05ed648e7b6626d06e08d8aec"
format_stamp: "Formatted at 2026-09-25 10:07:13 on dist-test-slave-f9rl"
server_key: "f5d60db3130042b7f2d22e0aeb8b598f"
server_key_iv: "aecd4b36cb34ef6c65b0027290b8a896"
server_key_version: "encryptionkey@0"
I20260925 10:07:13.636132 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "1cec78b05ed648e7b6626d06e08d8aec"
format_stamp: "Formatted at 2026-09-25 10:07:13 on dist-test-slave-f9rl"
server_key: "f5d60db3130042b7f2d22e0aeb8b598f"
server_key_iv: "aecd4b36cb34ef6c65b0027290b8a896"
server_key_version: "encryptionkey@0"
I20260925 10:07:13.638020 7742 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:07:13.639389 7757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:13.639926 7742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:13.640234 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "1cec78b05ed648e7b6626d06e08d8aec"
format_stamp: "Formatted at 2026-09-25 10:07:13 on dist-test-slave-f9rl"
server_key: "f5d60db3130042b7f2d22e0aeb8b598f"
server_key_iv: "aecd4b36cb34ef6c65b0027290b8a896"
server_key_version: "encryptionkey@0"
I20260925 10:07:13.640381 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:13.651887 7742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:13.655517 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:13.655728 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:13.660333 7742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34501
I20260925 10:07:13.660538 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34501 every 8 connection(s)
I20260925 10:07:13.660830 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:13.662446 7811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:13.663859 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7742
I20260925 10:07:13.664075 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:13.664299 19643 external_mini_cluster.cc:1428] Setting key dffc2799392a689dd8f80420c1a173a5
I20260925 10:07:13.667372 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec: Bootstrap starting.
I20260925 10:07:13.668222 7811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:13.668604 7811 log.cc:824] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:13.669505 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec: No bootstrap required, opened a new log
I20260925 10:07:13.672044 7811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } }
I20260925 10:07:13.672261 7811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:13.672323 7811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cec78b05ed648e7b6626d06e08d8aec, State: Initialized, Role: FOLLOWER
I20260925 10:07:13.672518 7811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } }
I20260925 10:07:13.672641 7811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:13.672694 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:13.672751 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:07:13.673617 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } }
I20260925 10:07:13.673763 7811 leader_election.cc:302] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cec78b05ed648e7b6626d06e08d8aec; no voters:
I20260925 10:07:13.673969 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:13.674031 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:13.674371 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [term 1 LEADER]: Becoming Leader. State: Replica: 1cec78b05ed648e7b6626d06e08d8aec, State: Running, Role: LEADER
I20260925 10:07:13.674603 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } }
I20260925 10:07:13.674634 7811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:13.675275 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cec78b05ed648e7b6626d06e08d8aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } } }
I20260925 10:07:13.675385 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:13.675635 7823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:13.676488 7823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:13.676702 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cec78b05ed648e7b6626d06e08d8aec. Latest consensus state: current_term: 1 leader_uuid: "1cec78b05ed648e7b6626d06e08d8aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cec78b05ed648e7b6626d06e08d8aec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34501 } } }
I20260925 10:07:13.676802 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:13.678399 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:13.665068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52381 (local address 127.0.0.1:34501)
0925 10:07:13.665417 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:13.665428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:13.665494 (+ 66us) server_negotiation.cc:407] Connection header received
0925 10:07:13.666230 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:13.666240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:13.666334 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:13.666526 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:13.667244 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:13.668607 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:13.669501 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:13.670320 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:13.670583 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:13.673772 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:13.673780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:13.673780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:13.673803 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:13.676337 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:13.676871 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:13.676875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:13.676875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:13.677027 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:13.677327 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:13.677328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:13.677329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:13.677690 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:13.677866 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:13.678032 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:07:13.678201 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":216,"threads_started":1}
I20260925 10:07:13.678490 7823 catalog_manager.cc:1347] Generated new cluster ID: 6dfdbf442cf1413ea0a122be0ef39ff3
I20260925 10:07:13.678530 7823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:13.679529 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:13.688737 7823 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:13.689347 7823 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:13.690207 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:13.681517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56863 (local address 127.0.0.1:34501)
0925 10:07:13.681563 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:13.681565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:13.681574 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:13.681596 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:13.681597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:13.681606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:13.681658 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:13.682369 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:13.683246 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:13.684308 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:13.684680 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:13.684768 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:13.686679 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:13.686681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:13.686681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:13.686691 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:13.688658 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:13.689169 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:13.689171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:13.689172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:13.689236 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:13.689497 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:13.689498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:13.689499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:13.689647 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:13.689743 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:13.689999 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:07:13.690110 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:07:13.699674 7823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cec78b05ed648e7b6626d06e08d8aec: Generated new TSK 0
I20260925 10:07:13.700001 7823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:13.703027 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7742
2026-09-25T10:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7842](info): set up 2 sockets
Sep 25 10:07:13 dist-test-slave-f9rl krb5kdc[7842](info): commencing operation
krb5kdc: starting...
W20260925 10:07:14.900198 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:14 dist-test-slave-f9rl krb5kdc[7842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:15.055755 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:15.184367 7855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:15.184773 7855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:15.184813 7855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:15.186872 7855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:15.186949 7855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:15.186968 7855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:15.186983 7855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:15.186997 7855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:15.188958 7855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:15.189278 7855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:15.189613 7855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:15.195936 7860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:15.196059 7861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:15.196194 7863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:15.196462 7855 server_base.cc:1034] running on GCE node
I20260925 10:07:15.196815 7855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:15.197218 7855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:15.198424 7855 hybrid_clock.cc:648] HybridClock initialized: now 1790330835198408 us; error 32 us; skew 500 ppm
Sep 25 10:07:15 dist-test-slave-f9rl krb5kdc[7842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:15.207381 7855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:15.208036 7855 webserver.cc:460] Webserver started at http://127.0.0.1:36109/ using document root <none> and password file <none>
I20260925 10:07:15.208335 7855 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:15.208417 7855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:15.208616 7855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:15.210067 7855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "05b3a7126ee64511b3229762c0d2d03f"
format_stamp: "Formatted at 2026-09-25 10:07:15 on dist-test-slave-f9rl"
server_key: "676c30993b8e7d6d83525d30c447cedd"
server_key_iv: "7b787645c02907b3af4922b0ecea8998"
server_key_version: "encryptionkey@0"
I20260925 10:07:15.210512 7855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "05b3a7126ee64511b3229762c0d2d03f"
format_stamp: "Formatted at 2026-09-25 10:07:15 on dist-test-slave-f9rl"
server_key: "676c30993b8e7d6d83525d30c447cedd"
server_key_iv: "7b787645c02907b3af4922b0ecea8998"
server_key_version: "encryptionkey@0"
I20260925 10:07:15.212448 7855 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:07:15.213973 7870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:15.214515 7855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:15.214804 7855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "05b3a7126ee64511b3229762c0d2d03f"
format_stamp: "Formatted at 2026-09-25 10:07:15 on dist-test-slave-f9rl"
server_key: "676c30993b8e7d6d83525d30c447cedd"
server_key_iv: "7b787645c02907b3af4922b0ecea8998"
server_key_version: "encryptionkey@0"
I20260925 10:07:15.215101 7855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:15.257062 7855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:15.259995 7855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:15.260206 7855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:15.265022 7855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36983
I20260925 10:07:15.265072 7922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36983 every 8 connection(s)
I20260925 10:07:15.265501 7855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:15.266638 7923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:15.269500 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f: Bootstrap starting.
I20260925 10:07:15.270251 7923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:15.270579 7923 log.cc:824] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:15.271626 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f: No bootstrap required, opened a new log
I20260925 10:07:15.273818 7923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } }
I20260925 10:07:15.274035 7923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:15.274081 7923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b3a7126ee64511b3229762c0d2d03f, State: Initialized, Role: FOLLOWER
I20260925 10:07:15.274272 7923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } }
I20260925 10:07:15.274374 7923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:15.274427 7923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:15.274542 7923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:15.275285 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7855
I20260925 10:07:15.275362 7923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } }
I20260925 10:07:15.275421 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:15.275499 7923 leader_election.cc:302] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b3a7126ee64511b3229762c0d2d03f; no voters:
I20260925 10:07:15.275621 19643 external_mini_cluster.cc:1428] Setting key 4d461ab311a45747a978771aee6de4f7
I20260925 10:07:15.275756 7923 leader_election.cc:288] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:15.275789 7926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:15.276171 7926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [term 1 LEADER]: Becoming Leader. State: Replica: 05b3a7126ee64511b3229762c0d2d03f, State: Running, Role: LEADER
I20260925 10:07:15.276202 7923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:15.276309 7926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } }
I20260925 10:07:15.277009 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b3a7126ee64511b3229762c0d2d03f. Latest consensus state: current_term: 1 leader_uuid: "05b3a7126ee64511b3229762c0d2d03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } } }
I20260925 10:07:15.277031 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b3a7126ee64511b3229762c0d2d03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b3a7126ee64511b3229762c0d2d03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36983 } } }
I20260925 10:07:15.277230 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:15.277520 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:15.277683 7938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:15.278491 7938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:15.282986 7938 catalog_manager.cc:1347] Generated new cluster ID: 0e1ad7f76b0e4ddb94b16a743f641f3b
I20260925 10:07:15.283078 7938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:15.283286 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:15.276539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53849 (local address 127.0.0.1:36983)
0925 10:07:15.277137 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:15.277148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:15.277210 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:07:15.277402 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:15.277411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:15.277518 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:15.277707 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:15.278402 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:15.279780 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:15.280765 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:15.281263 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:15.281415 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:15.282285 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:15.282291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:15.282291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:15.282533 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:15.282580 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:15.282598 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:15.282715 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:15.282926 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:07:15.283031 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":76,"threads_started":1}
I20260925 10:07:15.284423 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:15.299007 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:15.293930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52455 (local address 127.0.0.1:36983)
0925 10:07:15.293987 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:15.293990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:15.294166 (+ 176us) server_negotiation.cc:407] Connection header received
0925 10:07:15.294215 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:15.294217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:15.294230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:15.294310 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:15.294473 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:15.295401 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:15.296736 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:15.297118 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:15.297205 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:15.298452 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:15.298454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:15.298454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:15.298464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:15.298490 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:15.298495 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:15.298577 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:15.298787 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 10:07:15.298827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:07:15.330483 7938 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:15.331382 7938 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:15.347295 7938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b3a7126ee64511b3229762c0d2d03f: Generated new TSK 0
I20260925 10:07:15.347621 7938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:15.354136 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7855
2026-09-25T10:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1650 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:15 dist-test-slave-f9rl krb5kdc[7954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:15 dist-test-slave-f9rl krb5kdc[7954](info): set up 2 sockets
Sep 25 10:07:15 dist-test-slave-f9rl krb5kdc[7954](info): commencing operation
krb5kdc: starting...
W20260925 10:07:16.482914 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:16 dist-test-slave-f9rl krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:16.617012 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:16.708606 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:16.708874 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:16.708920 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:16.710147 7967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:16.710227 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:16.710251 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:16.710273 7967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:16.710304 7967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:16.711858 7967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:16.712190 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:16.712441 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:16.718042 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:16.718111 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:16.718154 7967 server_base.cc:1034] running on GCE node
W20260925 10:07:16.718111 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:16.718528 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:16.718861 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:16.720057 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790330836720018 us; error 51 us; skew 500 ppm
Sep 25 10:07:16 dist-test-slave-f9rl krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:16.723573 7967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:16.724231 7967 webserver.cc:460] Webserver started at http://127.0.0.1:41563/ using document root <none> and password file <none>
I20260925 10:07:16.724455 7967 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:16.724498 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:16.724596 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:16.725854 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "fe35c8f7a6f248b58da77c8c8ea4c760"
format_stamp: "Formatted at 2026-09-25 10:07:16 on dist-test-slave-f9rl"
server_key: "ff456f25197c80ba3ce8c177d756ba4e"
server_key_iv: "8690582c0bcedce00a64d7d8d160f6e8"
server_key_version: "encryptionkey@0"
I20260925 10:07:16.726198 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "fe35c8f7a6f248b58da77c8c8ea4c760"
format_stamp: "Formatted at 2026-09-25 10:07:16 on dist-test-slave-f9rl"
server_key: "ff456f25197c80ba3ce8c177d756ba4e"
server_key_iv: "8690582c0bcedce00a64d7d8d160f6e8"
server_key_version: "encryptionkey@0"
I20260925 10:07:16.727942 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:07:16.728772 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:16.728997 7967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:16.729095 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "fe35c8f7a6f248b58da77c8c8ea4c760"
format_stamp: "Formatted at 2026-09-25 10:07:16 on dist-test-slave-f9rl"
server_key: "ff456f25197c80ba3ce8c177d756ba4e"
server_key_iv: "8690582c0bcedce00a64d7d8d160f6e8"
server_key_version: "encryptionkey@0"
I20260925 10:07:16.729152 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:16.733924 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:16.736341 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:16.736486 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:16.740573 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33091
I20260925 10:07:16.740653 8034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33091 every 8 connection(s)
I20260925 10:07:16.741037 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:16.742434 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:16.742775 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 7967
I20260925 10:07:16.742966 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:16.743197 19643 external_mini_cluster.cc:1428] Setting key d56f450f3356aa9016c2eb5dfd7c9064
I20260925 10:07:16.746547 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:16.743985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59361 (local address 127.0.0.1:33091)
0925 10:07:16.744192 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:16.744203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:16.744242 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:07:16.744939 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:16.744949 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:16.744986 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:16.745157 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:16.745925 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:16.745928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:16.745929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:16.745933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:16.745979 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:16.745993 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:16.746090 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:16.746286 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:07:16.746331 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260925 10:07:16.747184 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760: Bootstrap starting.
I20260925 10:07:16.747814 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:16.747952 8035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:16.748243 8035 log.cc:824] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:16.749076 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760: No bootstrap required, opened a new log
I20260925 10:07:16.750689 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:16.749560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58273 (local address 127.0.0.1:33091)
0925 10:07:16.749608 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:16.749609 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:16.749758 (+ 149us) server_negotiation.cc:407] Connection header received
0925 10:07:16.749782 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:16.749783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:16.749787 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:16.749827 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:16.750413 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:16.750415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:16.750415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:16.750419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:16.750432 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:16.750436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:16.750491 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:16.750568 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:07:16.750579 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:07:16.751843 8035 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260925 10:07:16.751986 8035 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:16.752008 8035 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe35c8f7a6f248b58da77c8c8ea4c760, State: Initialized, Role: FOLLOWER
I20260925 10:07:16.752161 8035 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260925 10:07:16.752251 8035 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:16.752297 8035 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:16.752348 8035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:16.752938 8035 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260925 10:07:16.753073 8035 leader_election.cc:302] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe35c8f7a6f248b58da77c8c8ea4c760; no voters:
I20260925 10:07:16.753275 8035 leader_election.cc:288] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:16.753361 8045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:16.753582 8035 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:16.753616 8045 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [term 1 LEADER]: Becoming Leader. State: Replica: fe35c8f7a6f248b58da77c8c8ea4c760, State: Running, Role: LEADER
I20260925 10:07:16.753782 8045 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260925 10:07:16.754199 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe35c8f7a6f248b58da77c8c8ea4c760. Latest consensus state: current_term: 1 leader_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260925 10:07:16.754180 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe35c8f7a6f248b58da77c8c8ea4c760" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260925 10:07:16.754316 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:16.754324 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:16.754611 8054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:16.755276 8054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:16.756870 8054 catalog_manager.cc:1347] Generated new cluster ID: c1561af7d5a04261bfb4101815537220
I20260925 10:07:16.756934 8054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:16.761034 8054 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:16.761220 8054 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:16.765745 8054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe35c8f7a6f248b58da77c8c8ea4c760: Generated new TSK 0
I20260925 10:07:16.765969 8054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:16.772240 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 7967
2026-09-25T10:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:16 dist-test-slave-f9rl krb5kdc[8066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:16 dist-test-slave-f9rl krb5kdc[8066](info): set up 2 sockets
Sep 25 10:07:16 dist-test-slave-f9rl krb5kdc[8066](info): commencing operation
krb5kdc: starting...
W20260925 10:07:17.835922 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:17 dist-test-slave-f9rl krb5kdc[8066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:17.966902 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:18.048448 8079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:18.048729 8079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:18.048770 8079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:18.050088 8079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:18.050169 8079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:18.050196 8079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:18.050225 8079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:18.050256 8079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:18.051781 8079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:18.052081 8079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:18.052325 8079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:18.057338 8084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:18.057335 8087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:18.057507 8085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:18.057667 8079 server_base.cc:1034] running on GCE node
I20260925 10:07:18.057966 8079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:18.058243 8079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:18.059440 8079 hybrid_clock.cc:648] HybridClock initialized: now 1790330838059415 us; error 42 us; skew 500 ppm
Sep 25 10:07:18 dist-test-slave-f9rl krb5kdc[8066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:18.062306 8079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:18.062798 8079 webserver.cc:460] Webserver started at http://127.0.0.1:36011/ using document root <none> and password file <none>
I20260925 10:07:18.063040 8079 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:18.063088 8079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:18.063221 8079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:18.064343 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "bf9e4a2d1de242cdb4022aec85c31ce2"
format_stamp: "Formatted at 2026-09-25 10:07:18 on dist-test-slave-f9rl"
server_key: "3b167601e2a1aae4c7390445f21242be"
server_key_iv: "8984382365ae4aa690d4e22572757af3"
server_key_version: "encryptionkey@0"
I20260925 10:07:18.064729 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "bf9e4a2d1de242cdb4022aec85c31ce2"
format_stamp: "Formatted at 2026-09-25 10:07:18 on dist-test-slave-f9rl"
server_key: "3b167601e2a1aae4c7390445f21242be"
server_key_iv: "8984382365ae4aa690d4e22572757af3"
server_key_version: "encryptionkey@0"
I20260925 10:07:18.066303 8079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:18.067173 8094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:18.067385 8079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:18.067508 8079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "bf9e4a2d1de242cdb4022aec85c31ce2"
format_stamp: "Formatted at 2026-09-25 10:07:18 on dist-test-slave-f9rl"
server_key: "3b167601e2a1aae4c7390445f21242be"
server_key_iv: "8984382365ae4aa690d4e22572757af3"
server_key_version: "encryptionkey@0"
I20260925 10:07:18.067586 8079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:18.082185 8079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:18.084530 8079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:18.084703 8079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:18.089025 8079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37863
I20260925 10:07:18.089089 8146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37863 every 8 connection(s)
I20260925 10:07:18.089427 8079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:18.090294 8147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:18.092271 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8079
I20260925 10:07:18.092365 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:18.092571 19643 external_mini_cluster.cc:1428] Setting key 113c5c2bc88b80ceed132e6fd8386894
I20260925 10:07:18.092854 8147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2: Bootstrap starting.
I20260925 10:07:18.093782 8147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:18.094107 8147 log.cc:824] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:18.094874 8147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2: No bootstrap required, opened a new log
I20260925 10:07:18.096750 8147 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260925 10:07:18.096920 8147 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:18.096971 8147 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9e4a2d1de242cdb4022aec85c31ce2, State: Initialized, Role: FOLLOWER
I20260925 10:07:18.097159 8147 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260925 10:07:18.097259 8147 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:18.097302 8147 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:18.097352 8147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:18.097944 8147 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260925 10:07:18.098070 8147 leader_election.cc:302] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf9e4a2d1de242cdb4022aec85c31ce2; no voters:
I20260925 10:07:18.098258 8147 leader_election.cc:288] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:18.098296 8152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:18.098519 8147 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:18.098559 8152 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [term 1 LEADER]: Becoming Leader. State: Replica: bf9e4a2d1de242cdb4022aec85c31ce2, State: Running, Role: LEADER
I20260925 10:07:18.098703 8152 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260925 10:07:18.099140 8152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf9e4a2d1de242cdb4022aec85c31ce2. Latest consensus state: current_term: 1 leader_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260925 10:07:18.099161 8154 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9e4a2d1de242cdb4022aec85c31ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260925 10:07:18.099339 8154 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:18.099329 8152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:18.099740 8161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 10:07:18 dist-test-slave-f9rl krb5kdc[8066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:07:18.100590 8161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:18.102360 8161 catalog_manager.cc:1347] Generated new cluster ID: 48a6fc00f076409a903caaa6746ebb81
I20260925 10:07:18.102459 8161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:18.105067 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:18.093081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:42075 (local address 127.0.0.1:37863)
0925 10:07:18.093294 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:18.093303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:18.093338 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:07:18.094050 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:18.094057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:18.094112 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:18.094275 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:18.094967 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:18.095909 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:18.096823 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:18.097566 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:18.097703 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:18.100874 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:18.100878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:18.100878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:18.100892 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:18.103273 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:18.103744 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:18.103747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:18.103748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:18.103857 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:18.104122 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:18.104122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:18.104123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:18.104435 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:18.104580 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:18.104733 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:07:18.104836 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260925 10:07:18.106156 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:18.114535 8161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:18.115098 8161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:18.115998 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:18.107755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:51199 (local address 127.0.0.1:37863)
0925 10:07:18.107791 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:18.107793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:18.107843 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:07:18.107907 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:18.107908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:18.107918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:18.107977 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:18.108622 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:18.109458 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:18.110489 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:18.110846 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:18.110920 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:18.112480 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:18.112481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:18.112482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:18.112491 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:18.114293 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:18.114886 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:18.114887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:18.114888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:18.114947 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:18.115192 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:18.115193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:18.115194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:18.115319 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:18.115415 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:18.115837 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 10:07:18.115904 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:07:18.122258 8161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf9e4a2d1de242cdb4022aec85c31ce2: Generated new TSK 0
I20260925 10:07:18.122515 8161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:18.127776 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8079
W20260925 10:07:18.132470 8064 connection.cc:535] client connection to 127.0.0.1:37863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37863): Connection reset by peer (error 104)
2026-09-25T10:07:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:18 dist-test-slave-f9rl krb5kdc[8178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:18 dist-test-slave-f9rl krb5kdc[8178](info): set up 2 sockets
Sep 25 10:07:18 dist-test-slave-f9rl krb5kdc[8178](info): commencing operation
krb5kdc: starting...
W20260925 10:07:19.243741 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:19 dist-test-slave-f9rl krb5kdc[8178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:19.381276 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:19.470417 8191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:19.470669 8191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:19.470709 8191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:19.471907 8191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:19.471990 8191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:19.472024 8191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:19.472079 8191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:19.472100 8191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:19.473722 8191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:19.474002 8191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:19.474229 8191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:19.479794 8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:19.480288 8199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:19.480422 8191 server_base.cc:1034] running on GCE node
W20260925 10:07:19.480288 8197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:19.480746 8191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:19.481051 8191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:19.482209 8191 hybrid_clock.cc:648] HybridClock initialized: now 1790330839482199 us; error 35 us; skew 500 ppm
Sep 25 10:07:19 dist-test-slave-f9rl krb5kdc[8178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:19.485276 8191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:19.485833 8191 webserver.cc:460] Webserver started at http://127.0.0.1:35825/ using document root <none> and password file <none>
I20260925 10:07:19.486109 8191 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:19.486161 8191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:19.486351 8191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:19.487581 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "ec6072ea0c48458fabeb362f9e2f9461"
format_stamp: "Formatted at 2026-09-25 10:07:19 on dist-test-slave-f9rl"
server_key: "375a4c4118718564e92fd2c6ce844c09"
server_key_iv: "8470b10fa37fc6676fef491ed467ebcc"
server_key_version: "encryptionkey@0"
I20260925 10:07:19.487990 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "ec6072ea0c48458fabeb362f9e2f9461"
format_stamp: "Formatted at 2026-09-25 10:07:19 on dist-test-slave-f9rl"
server_key: "375a4c4118718564e92fd2c6ce844c09"
server_key_iv: "8470b10fa37fc6676fef491ed467ebcc"
server_key_version: "encryptionkey@0"
I20260925 10:07:19.489573 8191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:19.490537 8206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:19.490912 8191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:19.491044 8191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "ec6072ea0c48458fabeb362f9e2f9461"
format_stamp: "Formatted at 2026-09-25 10:07:19 on dist-test-slave-f9rl"
server_key: "375a4c4118718564e92fd2c6ce844c09"
server_key_iv: "8470b10fa37fc6676fef491ed467ebcc"
server_key_version: "encryptionkey@0"
I20260925 10:07:19.491137 8191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:19.510033 8191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:19.513604 8191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:19.513784 8191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:19.520069 8258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34567 every 8 connection(s)
I20260925 10:07:19.520150 8191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34567
I20260925 10:07:19.520798 8191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:19.522347 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:19.525493 8259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461: Bootstrap starting.
I20260925 10:07:19.526535 8259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:19.526885 8259 log.cc:824] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:19.527074 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8191
I20260925 10:07:19.527208 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:19.527420 19643 external_mini_cluster.cc:1428] Setting key 1d70666b325baf4ec305f8ece4ae6623
I20260925 10:07:19.527843 8259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461: No bootstrap required, opened a new log
I20260925 10:07:19.529876 8259 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } }
I20260925 10:07:19.530062 8259 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:19.530109 8259 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6072ea0c48458fabeb362f9e2f9461, State: Initialized, Role: FOLLOWER
I20260925 10:07:19.530241 8259 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } }
I20260925 10:07:19.530328 8259 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:19.530367 8259 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:19.530419 8259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:19.531162 8259 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } }
I20260925 10:07:19.531288 8259 leader_election.cc:302] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec6072ea0c48458fabeb362f9e2f9461; no voters:
I20260925 10:07:19.531484 8259 leader_election.cc:288] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:19.531507 8264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:19.531628 8264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [term 1 LEADER]: Becoming Leader. State: Replica: ec6072ea0c48458fabeb362f9e2f9461, State: Running, Role: LEADER
I20260925 10:07:19.531760 8259 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:19.531775 8264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } }
I20260925 10:07:19.532277 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec6072ea0c48458fabeb362f9e2f9461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } } }
I20260925 10:07:19.532372 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:19.532586 8266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6072ea0c48458fabeb362f9e2f9461. Latest consensus state: current_term: 1 leader_uuid: "ec6072ea0c48458fabeb362f9e2f9461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6072ea0c48458fabeb362f9e2f9461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34567 } } }
I20260925 10:07:19.532698 8266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:19.532877 8272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:19.533468 8272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:19.535481 8272 catalog_manager.cc:1347] Generated new cluster ID: 713651f6cce245d4aa986190dd8bd420
I20260925 10:07:19.535539 8272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:19.535707 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:19.528012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:50277 (local address 127.0.0.1:34567)
0925 10:07:19.528227 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:19.528237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:19.528273 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:07:19.529887 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:19.529896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:19.529957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:19.530136 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:19.530380 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:19.531866 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:19.533021 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:19.533704 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:19.533847 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:19.534882 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:19.534886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:19.534887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:19.535104 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:19.535144 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:19.535159 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:19.535248 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:19.535412 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:07:19.535443 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260925 10:07:19.536655 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:19.541045 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:19.538047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:36227 (local address 127.0.0.1:34567)
0925 10:07:19.538087 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:19.538089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:19.538237 (+ 148us) server_negotiation.cc:407] Connection header received
0925 10:07:19.538293 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:19.538295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:19.538306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:19.538374 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:19.538531 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:19.539020 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:19.539743 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:19.539969 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:19.540047 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:19.540807 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:19.540809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:19.540809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:19.540814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:19.540828 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:19.540832 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:19.540884 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:19.540945 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:07:19.540957 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:07:19.548466 8272 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:19.549185 8272 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:19.556708 8272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec6072ea0c48458fabeb362f9e2f9461: Generated new TSK 0
I20260925 10:07:19.556918 8272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:19.564847 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8191
W20260925 10:07:19.568728 8176 connection.cc:535] client connection to 127.0.0.1:34567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34567): Connection reset by peer (error 104)
2026-09-25T10:07:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:19 dist-test-slave-f9rl krb5kdc[8290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:19 dist-test-slave-f9rl krb5kdc[8290](info): set up 2 sockets
Sep 25 10:07:19 dist-test-slave-f9rl krb5kdc[8290](info): commencing operation
krb5kdc: starting...
W20260925 10:07:20.675647 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:20 dist-test-slave-f9rl krb5kdc[8290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:20.805136 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:20.889155 8303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:20.889379 8303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:20.889405 8303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:20.890718 8303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:20.890770 8303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:20.890785 8303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:20.890798 8303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:20.890810 8303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:20.892283 8303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:20.892501 8303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:20.892688 8303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:20.898160 8308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:20.898204 8303 server_base.cc:1034] running on GCE node
W20260925 10:07:20.898168 8309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:20.898190 8311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:20.898722 8303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:20.899088 8303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:20.900267 8303 hybrid_clock.cc:648] HybridClock initialized: now 1790330840900239 us; error 52 us; skew 500 ppm
Sep 25 10:07:20 dist-test-slave-f9rl krb5kdc[8290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:20.907910 8303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:20.908474 8303 webserver.cc:460] Webserver started at http://127.0.0.1:33997/ using document root <none> and password file <none>
I20260925 10:07:20.908731 8303 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:20.908804 8303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:20.908968 8303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:20.910218 8303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "29fe599ea39445de9bfff79c48cdf979"
format_stamp: "Formatted at 2026-09-25 10:07:20 on dist-test-slave-f9rl"
server_key: "4c2663805060bb5a907953b060b27021"
server_key_iv: "aaef79cf974cf5e5e3d4b6d5c5d45a9e"
server_key_version: "encryptionkey@0"
I20260925 10:07:20.910597 8303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "29fe599ea39445de9bfff79c48cdf979"
format_stamp: "Formatted at 2026-09-25 10:07:20 on dist-test-slave-f9rl"
server_key: "4c2663805060bb5a907953b060b27021"
server_key_iv: "aaef79cf974cf5e5e3d4b6d5c5d45a9e"
server_key_version: "encryptionkey@0"
I20260925 10:07:20.912089 8303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:20.913075 8318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:20.913493 8303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:20.913625 8303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "29fe599ea39445de9bfff79c48cdf979"
format_stamp: "Formatted at 2026-09-25 10:07:20 on dist-test-slave-f9rl"
server_key: "4c2663805060bb5a907953b060b27021"
server_key_iv: "aaef79cf974cf5e5e3d4b6d5c5d45a9e"
server_key_version: "encryptionkey@0"
I20260925 10:07:20.913710 8303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:20.917325 8303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:20.919752 8303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:20.919924 8303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:20.924383 8303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40051
I20260925 10:07:20.924506 8370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40051 every 8 connection(s)
I20260925 10:07:20.924813 8303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:20.925864 8371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:20.928303 8371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: Bootstrap starting.
I20260925 10:07:20.928848 8371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:20.929083 8371 log.cc:824] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:20.929886 8371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: No bootstrap required, opened a new log
I20260925 10:07:20.930292 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8303
I20260925 10:07:20.930444 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:20.930644 19643 external_mini_cluster.cc:1428] Setting key 660c49aa7a4a9170ba53799a4a985a0b
I20260925 10:07:20.932425 8371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } }
I20260925 10:07:20.932579 8371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:20.932605 8371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fe599ea39445de9bfff79c48cdf979, State: Initialized, Role: FOLLOWER
I20260925 10:07:20.932862 8371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } }
I20260925 10:07:20.933049 8371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:20.933135 8371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:20.933198 8371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:20.934198 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:20.931287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:32941 (local address 127.0.0.1:40051)
0925 10:07:20.931483 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:20.931492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:20.931606 (+ 114us) server_negotiation.cc:407] Connection header received
0925 10:07:20.932402 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:20.932410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:20.932441 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:20.932684 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:20.933641 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:20.933645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:20.933645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:20.933649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:20.933693 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:20.933705 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:20.933769 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:20.933871 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:07:20.933934 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260925 10:07:20.934648 8371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } }
I20260925 10:07:20.934777 8371 leader_election.cc:302] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29fe599ea39445de9bfff79c48cdf979; no voters:
I20260925 10:07:20.934960 8371 leader_election.cc:288] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:20.935029 8376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:20.935252 8371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:20.935251 8376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [term 1 LEADER]: Becoming Leader. State: Replica: 29fe599ea39445de9bfff79c48cdf979, State: Running, Role: LEADER
I20260925 10:07:20.935495 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:20.935491 8376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } }
I20260925 10:07:20.936033 8378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29fe599ea39445de9bfff79c48cdf979. Latest consensus state: current_term: 1 leader_uuid: "29fe599ea39445de9bfff79c48cdf979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } } }
I20260925 10:07:20.936129 8378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:20.936434 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29fe599ea39445de9bfff79c48cdf979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fe599ea39445de9bfff79c48cdf979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40051 } } }
I20260925 10:07:20.936559 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979 [sys.catalog]: This master's current role is: LEADER
W20260925 10:07:20.937459 8393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:20.937530 8393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:20.937584 8390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:20.937765 8390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:20.938926 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:20.937410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:39737 (local address 127.0.0.1:40051)
0925 10:07:20.937471 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:20.937473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:20.937599 (+ 126us) server_negotiation.cc:407] Connection header received
0925 10:07:20.937627 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:20.937627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:20.937634 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:20.937699 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:20.938479 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:20.938480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:20.938481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:20.938485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:20.938501 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:20.938507 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:20.938558 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:20.938833 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:07:20.938850 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:07:20.939611 8390 catalog_manager.cc:1347] Generated new cluster ID: a6306d3a3b274e04aed3c75a0ab6e57e
I20260925 10:07:20.939669 8390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:20.977926 8390 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:20.978163 8390 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:20.987337 8390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29fe599ea39445de9bfff79c48cdf979: Generated new TSK 0
I20260925 10:07:20.987600 8390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:20.997758 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8303
2026-09-25T10:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:21 dist-test-slave-f9rl krb5kdc[8402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:21 dist-test-slave-f9rl krb5kdc[8402](info): set up 2 sockets
Sep 25 10:07:21 dist-test-slave-f9rl krb5kdc[8402](info): commencing operation
krb5kdc: starting...
W20260925 10:07:22.091361 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:22.221244 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:22.306706 8415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:22.306999 8415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:22.307042 8415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:22.308207 8415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:22.308287 8415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:22.308314 8415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:22.308353 8415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:22.308377 8415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:22.309896 8415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:22.310181 8415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:22.310423 8415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:22.315657 8423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:22.315838 8415 server_base.cc:1034] running on GCE node
W20260925 10:07:22.315629 8420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:22.315629 8421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:22.316205 8415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:22.316576 8415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:22.317749 8415 hybrid_clock.cc:648] HybridClock initialized: now 1790330842317696 us; error 76 us; skew 500 ppm
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:22.320529 8415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:22.321040 8415 webserver.cc:460] Webserver started at http://127.0.0.1:37435/ using document root <none> and password file <none>
I20260925 10:07:22.321285 8415 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:22.321357 8415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:22.321533 8415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:22.322676 8415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "0d5e04f2ddb64b0280562dba219adf62"
format_stamp: "Formatted at 2026-09-25 10:07:22 on dist-test-slave-f9rl"
server_key: "c1fcd451a08ff234ba638ef78fab7be4"
server_key_iv: "ecc39f1b960faaedddb436317ec0c529"
server_key_version: "encryptionkey@0"
I20260925 10:07:22.323060 8415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "0d5e04f2ddb64b0280562dba219adf62"
format_stamp: "Formatted at 2026-09-25 10:07:22 on dist-test-slave-f9rl"
server_key: "c1fcd451a08ff234ba638ef78fab7be4"
server_key_iv: "ecc39f1b960faaedddb436317ec0c529"
server_key_version: "encryptionkey@0"
I20260925 10:07:22.324453 8415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:22.325296 8430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:22.325562 8415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:22.325677 8415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "0d5e04f2ddb64b0280562dba219adf62"
format_stamp: "Formatted at 2026-09-25 10:07:22 on dist-test-slave-f9rl"
server_key: "c1fcd451a08ff234ba638ef78fab7be4"
server_key_iv: "ecc39f1b960faaedddb436317ec0c529"
server_key_version: "encryptionkey@0"
I20260925 10:07:22.325763 8415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:22.345295 8415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:22.347926 8415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:22.348114 8415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:22.353302 8415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34733
I20260925 10:07:22.353391 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34733 every 8 connection(s)
I20260925 10:07:22.353855 8415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:22.354941 8483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:22.357537 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8415
I20260925 10:07:22.357636 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62: Bootstrap starting.
I20260925 10:07:22.357694 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:22.357910 19643 external_mini_cluster.cc:1428] Setting key ebd6fe7b8aa5d81e9049a4dda58151ce
I20260925 10:07:22.358445 8483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:22.358824 8483 log.cc:824] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:22.359717 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62: No bootstrap required, opened a new log
I20260925 10:07:22.362334 8483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260925 10:07:22.362555 8483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:22.362596 8483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5e04f2ddb64b0280562dba219adf62, State: Initialized, Role: FOLLOWER
I20260925 10:07:22.362730 8483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260925 10:07:22.362804 8483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:22.362838 8483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:22.362885 8483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:22.363572 8483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260925 10:07:22.363711 8483 leader_election.cc:302] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d5e04f2ddb64b0280562dba219adf62; no voters:
I20260925 10:07:22.363972 8483 leader_election.cc:288] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:22.363997 8488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:22.364110 8488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [term 1 LEADER]: Becoming Leader. State: Replica: 0d5e04f2ddb64b0280562dba219adf62, State: Running, Role: LEADER
I20260925 10:07:22.364200 8488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260925 10:07:22.364269 8483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:22.364543 8488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d5e04f2ddb64b0280562dba219adf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260925 10:07:22.364710 8488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:22.364940 8488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5e04f2ddb64b0280562dba219adf62. Latest consensus state: current_term: 1 leader_uuid: "0d5e04f2ddb64b0280562dba219adf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5e04f2ddb64b0280562dba219adf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260925 10:07:22.365077 8488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:22.365510 8495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:22.366166 8495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:07:22.368216 8495 catalog_manager.cc:1347] Generated new cluster ID: b4aafd9071214e5a81aff3a278a6b554
I20260925 10:07:22.368276 8495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:22.371878 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:22.358542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:47239 (local address 127.0.0.1:34733)
0925 10:07:22.358761 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:22.358770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:22.358798 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:07:22.359515 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:22.359523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:22.359581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:22.359711 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:22.360752 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:22.361883 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:22.363074 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:22.363609 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:22.363789 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:22.367285 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:22.367290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:22.367290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:22.367303 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:22.369625 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:22.370186 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:22.370190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:22.370190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:22.370313 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:22.370595 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:22.370596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:22.370597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:22.370954 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:22.371108 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:22.371459 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 10:07:22.371621 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260925 10:07:22.373138 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:22.374766 8495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:22.375329 8495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:22.383713 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:22.374833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:35999 (local address 127.0.0.1:34733)
0925 10:07:22.374898 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:22.374900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:22.375173 (+ 273us) server_negotiation.cc:407] Connection header received
0925 10:07:22.375283 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:22.375284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:22.375322 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:22.375369 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:22.376150 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:22.377057 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:22.378315 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:22.378661 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:22.378707 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:22.380465 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:22.380466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:22.380467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:22.380474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:22.382214 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:22.382712 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:22.382713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:22.382713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:22.382774 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:22.383043 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:22.383044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:22.383045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:22.383187 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:22.383273 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:22.383530 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:07:22.383603 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:07:22.392145 8495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d5e04f2ddb64b0280562dba219adf62: Generated new TSK 0
I20260925 10:07:22.392390 8495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:22.400763 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8415
W20260925 10:07:22.405682 8400 connection.cc:535] client connection to 127.0.0.1:34733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34733): Connection reset by peer (error 104)
2026-09-25T10:07:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8514](info): set up 2 sockets
Sep 25 10:07:22 dist-test-slave-f9rl krb5kdc[8514](info): commencing operation
krb5kdc: starting...
W20260925 10:07:23.503049 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:23 dist-test-slave-f9rl krb5kdc[8514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:23.636772 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:23.718307 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:23.718611 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:23.718669 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:23.719964 8527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:23.720045 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:23.720070 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:23.720096 8527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:23.720126 8527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:23.721719 8527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:23.722021 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:23.722262 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:07:23.727686 8527 server_base.cc:1034] running on GCE node
W20260925 10:07:23.727661 8533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:23.727626 8532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:23.727646 8535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:23.728067 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:23.728325 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:23.729542 8527 hybrid_clock.cc:648] HybridClock initialized: now 1790330843729528 us; error 32 us; skew 500 ppm
Sep 25 10:07:23 dist-test-slave-f9rl krb5kdc[8514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:23.732092 8527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:23.732504 8527 webserver.cc:460] Webserver started at http://127.0.0.1:35563/ using document root <none> and password file <none>
I20260925 10:07:23.732744 8527 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:23.732815 8527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:23.732972 8527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:23.734134 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "77b053a7fb1d4e45bfca88a941cb4cd2"
format_stamp: "Formatted at 2026-09-25 10:07:23 on dist-test-slave-f9rl"
server_key: "fe4935e1f6666d179a47b4f6b123e709"
server_key_iv: "faa8d4082dcec04bf9fc81c0e7758617"
server_key_version: "encryptionkey@0"
I20260925 10:07:23.734515 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "77b053a7fb1d4e45bfca88a941cb4cd2"
format_stamp: "Formatted at 2026-09-25 10:07:23 on dist-test-slave-f9rl"
server_key: "fe4935e1f6666d179a47b4f6b123e709"
server_key_iv: "faa8d4082dcec04bf9fc81c0e7758617"
server_key_version: "encryptionkey@0"
I20260925 10:07:23.735905 8527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:23.736732 8542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:23.737051 8527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:23.737213 8527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "77b053a7fb1d4e45bfca88a941cb4cd2"
format_stamp: "Formatted at 2026-09-25 10:07:23 on dist-test-slave-f9rl"
server_key: "fe4935e1f6666d179a47b4f6b123e709"
server_key_iv: "faa8d4082dcec04bf9fc81c0e7758617"
server_key_version: "encryptionkey@0"
I20260925 10:07:23.737309 8527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:23.770738 8527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:23.773103 8527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:23.773280 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:23.777416 8527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43559
I20260925 10:07:23.777583 8594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43559 every 8 connection(s)
I20260925 10:07:23.777915 8527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:23.778865 8595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:23.781567 8595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2: Bootstrap starting.
I20260925 10:07:23.782289 8595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:23.782428 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8527
I20260925 10:07:23.782574 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:23.782703 8595 log.cc:824] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:23.782797 19643 external_mini_cluster.cc:1428] Setting key d4631fcbdc4c473db06d9edc9b09cd23
I20260925 10:07:23.783738 8595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2: No bootstrap required, opened a new log
I20260925 10:07:23.787289 8595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260925 10:07:23.787475 8595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:23.787501 8595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b053a7fb1d4e45bfca88a941cb4cd2, State: Initialized, Role: FOLLOWER
I20260925 10:07:23.787679 8595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260925 10:07:23.787767 8595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:23.787815 8595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:23.787935 8595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:23.788902 8595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260925 10:07:23.789063 8595 leader_election.cc:302] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77b053a7fb1d4e45bfca88a941cb4cd2; no voters:
I20260925 10:07:23.789288 8595 leader_election.cc:288] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:23.789345 8600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:23.789614 8600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 77b053a7fb1d4e45bfca88a941cb4cd2, State: Running, Role: LEADER
I20260925 10:07:23.789718 8595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:23.789742 8600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260925 10:07:23.790266 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b053a7fb1d4e45bfca88a941cb4cd2. Latest consensus state: current_term: 1 leader_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260925 10:07:23.790467 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:23.790356 8601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b053a7fb1d4e45bfca88a941cb4cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260925 10:07:23.790735 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:23.783532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:48089 (local address 127.0.0.1:43559)
0925 10:07:23.783741 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:23.783752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:23.783798 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:07:23.784426 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:23.784435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:23.784519 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:23.784694 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:23.784915 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:23.786702 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:23.787851 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:23.788552 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:23.788691 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:23.789628 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:23.789632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:23.789633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:23.789870 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:23.789915 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:23.789933 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:23.790044 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:23.790466 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 10:07:23.790499 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260925 10:07:23.790812 8601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:23.791335 8609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:23.791914 8609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:23.792074 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:23.793746 8609 catalog_manager.cc:1347] Generated new cluster ID: 6fde85d7acd6443bb9f7309eea079674
I20260925 10:07:23.793810 8609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:23.797945 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:23.794076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:49613 (local address 127.0.0.1:43559)
0925 10:07:23.794114 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:23.794116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:23.794126 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:23.794185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:23.794186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:23.794196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:23.794257 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:23.794411 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:23.795158 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:23.796297 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:23.796644 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:23.796722 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:23.797675 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:23.797676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:23.797676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:23.797681 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:23.797697 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:23.797702 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:23.797771 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:23.797853 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:07:23.797865 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:07:23.801463 8609 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:23.802024 8609 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:23.810758 8609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77b053a7fb1d4e45bfca88a941cb4cd2: Generated new TSK 0
I20260925 10:07:23.810985 8609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:23.815580 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8527
W20260925 10:07:23.820178 8512 connection.cc:535] client connection to 127.0.0.1:43559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43559): Connection reset by peer (error 104)
2026-09-25T10:07:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:23 dist-test-slave-f9rl krb5kdc[8627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:23 dist-test-slave-f9rl krb5kdc[8627](info): set up 2 sockets
Sep 25 10:07:23 dist-test-slave-f9rl krb5kdc[8627](info): commencing operation
krb5kdc: starting...
W20260925 10:07:24.895416 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:24 dist-test-slave-f9rl krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:07:25.021237 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:25.107143 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:25.107376 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:25.107402 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:25.108608 8640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:25.108654 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:25.108668 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:25.108680 8640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:25.108693 8640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:25.110340 8640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:25.110579 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:25.110767 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:25.115957 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:25.115983 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:25.116004 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:25.116374 8640 server_base.cc:1034] running on GCE node
I20260925 10:07:25.116706 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:25.117027 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:25.118192 8640 hybrid_clock.cc:648] HybridClock initialized: now 1790330845118158 us; error 48 us; skew 500 ppm
Sep 25 10:07:25 dist-test-slave-f9rl krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:25.121009 8640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:07:25.121618 8640 webserver.cc:460] Webserver started at http://127.0.0.1:45969/ using document root <none> and password file <none>
I20260925 10:07:25.121884 8640 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:25.121959 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:25.122118 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:25.123238 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "69cbde00ca0c40e7977bcad6aba6b400"
format_stamp: "Formatted at 2026-09-25 10:07:25 on dist-test-slave-f9rl"
server_key: "c00977c66b293ca2db1bcae88fb0dca8"
server_key_iv: "e97aaf7c6769fe5dd5c03ab7d236ed11"
server_key_version: "encryptionkey@0"
I20260925 10:07:25.123667 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "69cbde00ca0c40e7977bcad6aba6b400"
format_stamp: "Formatted at 2026-09-25 10:07:25 on dist-test-slave-f9rl"
server_key: "c00977c66b293ca2db1bcae88fb0dca8"
server_key_iv: "e97aaf7c6769fe5dd5c03ab7d236ed11"
server_key_version: "encryptionkey@0"
I20260925 10:07:25.125145 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:25.126101 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:25.126348 8640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:07:25.126457 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "69cbde00ca0c40e7977bcad6aba6b400"
format_stamp: "Formatted at 2026-09-25 10:07:25 on dist-test-slave-f9rl"
server_key: "c00977c66b293ca2db1bcae88fb0dca8"
server_key_iv: "e97aaf7c6769fe5dd5c03ab7d236ed11"
server_key_version: "encryptionkey@0"
I20260925 10:07:25.126559 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:25.130414 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:25.132803 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:25.132956 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:25.137372 8640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44629
I20260925 10:07:25.137490 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44629 every 8 connection(s)
I20260925 10:07:25.137871 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:25.138883 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:25.141219 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400: Bootstrap starting.
I20260925 10:07:25.141881 8708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:25.142175 8708 log.cc:824] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:25.142961 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400: No bootstrap required, opened a new log
I20260925 10:07:25.144687 8708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260925 10:07:25.144837 8708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:25.144861 8708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69cbde00ca0c40e7977bcad6aba6b400, State: Initialized, Role: FOLLOWER
I20260925 10:07:25.145028 8708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260925 10:07:25.145092 8708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:25.145115 8708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:25.145167 8708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:25.145874 8708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260925 10:07:25.145968 8708 leader_election.cc:302] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69cbde00ca0c40e7977bcad6aba6b400; no voters:
I20260925 10:07:25.146186 8708 leader_election.cc:288] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:25.146234 8711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:25.146437 8711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [term 1 LEADER]: Becoming Leader. State: Replica: 69cbde00ca0c40e7977bcad6aba6b400, State: Running, Role: LEADER
I20260925 10:07:25.146584 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8640
I20260925 10:07:25.146606 8711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260925 10:07:25.146721 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:25.146942 19643 external_mini_cluster.cc:1428] Setting key ea235dec41031688f131e0c2a59af682
I20260925 10:07:25.146473 8708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:25.147038 8712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69cbde00ca0c40e7977bcad6aba6b400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260925 10:07:25.147114 8712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:25.147418 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69cbde00ca0c40e7977bcad6aba6b400. Latest consensus state: current_term: 1 leader_uuid: "69cbde00ca0c40e7977bcad6aba6b400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cbde00ca0c40e7977bcad6aba6b400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260925 10:07:25.147433 8716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:25.148022 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:25.148694 8716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:25.150022 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:25.147593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:45577 (local address 127.0.0.1:44629)
0925 10:07:25.147826 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:25.147849 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:25.147918 (+ 69us) server_negotiation.cc:407] Connection header received
0925 10:07:25.148116 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:25.148125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:25.148178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:25.148657 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:25.149519 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:25.149523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:25.149523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:25.149527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:25.149571 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:25.149586 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:25.149654 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:25.149776 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:07:25.149824 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260925 10:07:25.150513 8716 catalog_manager.cc:1347] Generated new cluster ID: f00198d05a6a4c95b517e385ffbfa454
I20260925 10:07:25.150684 8716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:25.151011 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:25.162660 8716 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:25.162858 8716 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:25.163755 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:25.162396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:59433 (local address 127.0.0.1:44629)
0925 10:07:25.162468 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:25.162471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:25.162611 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:07:25.162728 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:25.162729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:25.162736 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:25.162809 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:25.163527 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:25.163528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:25.163528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:25.163533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:25.163551 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:25.163556 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:25.163618 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:25.163672 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:07:25.163686 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:07:25.167608 8716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69cbde00ca0c40e7977bcad6aba6b400: Generated new TSK 0
I20260925 10:07:25.167810 8716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:25.175937 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8640
2026-09-25T10:07:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:25 dist-test-slave-f9rl krb5kdc[8740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:25 dist-test-slave-f9rl krb5kdc[8740](info): set up 2 sockets
Sep 25 10:07:25 dist-test-slave-f9rl krb5kdc[8740](info): commencing operation
krb5kdc: starting...
W20260925 10:07:26.239014 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:26.376561 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34677
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:26.463034 8753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:26.463259 8753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:26.463284 8753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:26.464468 8753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:26.464525 8753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:26.464540 8753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:26.464553 8753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:26.464566 8753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:26.466148 8753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:34677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:26.466409 8753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:26.466650 8753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:26.471897 8759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:26.471912 8761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:26.471942 8753 server_base.cc:1034] running on GCE node
W20260925 10:07:26.471912 8758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:26.472353 8753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:26.472702 8753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:26.473893 8753 hybrid_clock.cc:648] HybridClock initialized: now 1790330846473863 us; error 32 us; skew 500 ppm
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330846, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:26.482264 8753 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:26.482805 8753 webserver.cc:460] Webserver started at http://127.19.46.254:39195/ using document root <none> and password file <none>
I20260925 10:07:26.483048 8753 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:26.483098 8753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:26.483193 8753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:26.484391 8753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "234a46e4b06946a0803229d6915fc57f"
format_stamp: "Formatted at 2026-09-25 10:07:26 on dist-test-slave-f9rl"
server_key: "cb3b7598589d3998278d704d449e947b"
server_key_iv: "79776a1938a70a371bffc2fc191bc600"
server_key_version: "encryptionkey@0"
I20260925 10:07:26.484781 8753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "234a46e4b06946a0803229d6915fc57f"
format_stamp: "Formatted at 2026-09-25 10:07:26 on dist-test-slave-f9rl"
server_key: "cb3b7598589d3998278d704d449e947b"
server_key_iv: "79776a1938a70a371bffc2fc191bc600"
server_key_version: "encryptionkey@0"
I20260925 10:07:26.486346 8753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:26.487215 8768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:26.487406 8753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:26.487514 8753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "234a46e4b06946a0803229d6915fc57f"
format_stamp: "Formatted at 2026-09-25 10:07:26 on dist-test-slave-f9rl"
server_key: "cb3b7598589d3998278d704d449e947b"
server_key_iv: "79776a1938a70a371bffc2fc191bc600"
server_key_version: "encryptionkey@0"
I20260925 10:07:26.487589 8753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:26.508201 8753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:26.510489 8753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:26.510654 8753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:26.514935 8753 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:34677
I20260925 10:07:26.514981 8820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:34677 every 8 connection(s)
I20260925 10:07:26.515362 8753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:26.516306 8821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:26.518836 8821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f: Bootstrap starting.
I20260925 10:07:26.519471 8821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:26.519773 8821 log.cc:824] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:26.520520 8821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f: No bootstrap required, opened a new log
I20260925 10:07:26.522251 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8753
I20260925 10:07:26.522420 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:26.522321 8821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } }
I20260925 10:07:26.522557 8821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:26.522609 8821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234a46e4b06946a0803229d6915fc57f, State: Initialized, Role: FOLLOWER
I20260925 10:07:26.522650 19643 external_mini_cluster.cc:1428] Setting key e1115fb272b713b20da75a676eb4be51
I20260925 10:07:26.522768 8821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } }
I20260925 10:07:26.522864 8821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:26.522902 8821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:26.522970 8821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:26.524156 8821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } }
I20260925 10:07:26.524251 8821 leader_election.cc:302] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 234a46e4b06946a0803229d6915fc57f; no voters:
I20260925 10:07:26.524444 8821 leader_election.cc:288] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:26.524494 8826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:26.524712 8826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [term 1 LEADER]: Becoming Leader. State: Replica: 234a46e4b06946a0803229d6915fc57f, State: Running, Role: LEADER
I20260925 10:07:26.524717 8821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:26.524909 8826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } }
I20260925 10:07:26.525300 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234a46e4b06946a0803229d6915fc57f. Latest consensus state: current_term: 1 leader_uuid: "234a46e4b06946a0803229d6915fc57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } } }
I20260925 10:07:26.525429 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:26.525314 8827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "234a46e4b06946a0803229d6915fc57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234a46e4b06946a0803229d6915fc57f" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 34677 } } }
I20260925 10:07:26.525642 8827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:26.525734 8834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:26.526335 8834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:26.528002 8834 catalog_manager.cc:1347] Generated new cluster ID: 6cea076ad5284215a76511513cf6cb32
I20260925 10:07:26.528064 8834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:26.539106 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:26.523214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53415 (local address 127.19.46.254:34677)
0925 10:07:26.523437 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:26.523447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:26.523491 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:07:26.523636 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:26.523645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:26.523716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:26.523896 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:26.524814 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:26.526561 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:26.527761 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:26.528320 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:26.528405 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:26.531431 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:26.531435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:26.531436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:26.531450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:26.536988 (+ 5538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:26.537604 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:26.537608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:26.537608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:26.537765 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:26.538040 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:26.538041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:26.538042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:26.538405 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:26.538590 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:26.538738 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:07:26.538846 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260925 10:07:26.540514 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:26.544900 8834 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:26.545634 8834 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:26.550884 8834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 234a46e4b06946a0803229d6915fc57f: Generated new TSK 0
I20260925 10:07:26.551115 8834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:26.552811 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:26.544643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48005 (local address 127.19.46.254:34677)
0925 10:07:26.544685 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:26.544688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:26.544995 (+ 307us) server_negotiation.cc:407] Connection header received
0925 10:07:26.545086 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:26.545088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:26.545099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:26.545166 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:26.545886 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:26.546688 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:26.547722 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:26.548069 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:26.548134 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:26.549692 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:26.549693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:26.549694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:26.549702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:26.551536 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:26.551985 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:26.551986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:26.551986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:26.552043 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:26.552285 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:26.552286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:26.552286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:26.552418 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:26.552516 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:26.552694 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:07:26.552747 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:07:26.555295 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8753
W20260925 10:07:26.559710 8738 connection.cc:535] client connection to 127.19.46.254:34677 recv error: Network error: failed to read from TLS socket (remote: 127.19.46.254:34677): Connection reset by peer (error 104)
2026-09-25T10:07:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8853](info): set up 2 sockets
Sep 25 10:07:26 dist-test-slave-f9rl krb5kdc[8853](info): commencing operation
krb5kdc: starting...
W20260925 10:07:27.639663 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:27 dist-test-slave-f9rl krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:27.770395 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46217
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:27.854878 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:27.855194 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:27.855253 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:27.856833 8866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:27.856967 8866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:27.856997 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:27.857017 8866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:27.857034 8866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:27.858886 8866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:46217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:27.859242 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:27.859513 8866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:27.865254 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:27.865257 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:27.865257 8866 server_base.cc:1034] running on GCE node
W20260925 10:07:27.865254 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:27.865918 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:27.866223 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:27.867409 8866 hybrid_clock.cc:648] HybridClock initialized: now 1790330847867363 us; error 67 us; skew 500 ppm
Sep 25 10:07:27 dist-test-slave-f9rl krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:27.870566 8866 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:27.871110 8866 webserver.cc:460] Webserver started at http://127.19.46.254:43717/ using document root <none> and password file <none>
I20260925 10:07:27.871379 8866 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:27.871433 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:27.871569 8866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:27.872730 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "25d484c8930c4042ab7b4b15fc3afa05"
format_stamp: "Formatted at 2026-09-25 10:07:27 on dist-test-slave-f9rl"
server_key: "d468f6ca7b88c236dac04af4538a2ce0"
server_key_iv: "5347da07d89a425f3a29fae5fdbb5c2e"
server_key_version: "encryptionkey@0"
I20260925 10:07:27.873123 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "25d484c8930c4042ab7b4b15fc3afa05"
format_stamp: "Formatted at 2026-09-25 10:07:27 on dist-test-slave-f9rl"
server_key: "d468f6ca7b88c236dac04af4538a2ce0"
server_key_iv: "5347da07d89a425f3a29fae5fdbb5c2e"
server_key_version: "encryptionkey@0"
I20260925 10:07:27.874814 8866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:07:27.875824 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:27.876134 8866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:27.876253 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "25d484c8930c4042ab7b4b15fc3afa05"
format_stamp: "Formatted at 2026-09-25 10:07:27 on dist-test-slave-f9rl"
server_key: "d468f6ca7b88c236dac04af4538a2ce0"
server_key_iv: "5347da07d89a425f3a29fae5fdbb5c2e"
server_key_version: "encryptionkey@0"
I20260925 10:07:27.876355 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:27.899871 8866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:27.902681 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:27.902848 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:27.907667 8866 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:46217
I20260925 10:07:27.907707 8933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:46217 every 8 connection(s)
I20260925 10:07:27.908118 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:27.909147 8934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:27.911455 8934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05: Bootstrap starting.
I20260925 10:07:27.912009 8934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:27.912240 8934 log.cc:824] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:27.913048 8934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05: No bootstrap required, opened a new log
I20260925 10:07:27.914904 8934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } }
I20260925 10:07:27.915062 8934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:27.915086 8934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d484c8930c4042ab7b4b15fc3afa05, State: Initialized, Role: FOLLOWER
I20260925 10:07:27.915196 8934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } }
I20260925 10:07:27.915280 8934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:27.915331 8934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:27.915416 8934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:27.916401 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8866
I20260925 10:07:27.916451 8934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } }
I20260925 10:07:27.916539 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:27.916570 8934 leader_election.cc:302] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d484c8930c4042ab7b4b15fc3afa05; no voters:
I20260925 10:07:27.916719 19643 external_mini_cluster.cc:1428] Setting key fe42dce051a2e81cf0ea60de79a006ca
I20260925 10:07:27.916775 8934 leader_election.cc:288] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:27.916929 8937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:27.917047 8934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:27.917140 8937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [term 1 LEADER]: Becoming Leader. State: Replica: 25d484c8930c4042ab7b4b15fc3afa05, State: Running, Role: LEADER
I20260925 10:07:27.917276 8937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } }
I20260925 10:07:27.917845 8938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d484c8930c4042ab7b4b15fc3afa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } } }
I20260925 10:07:27.917843 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d484c8930c4042ab7b4b15fc3afa05. Latest consensus state: current_term: 1 leader_uuid: "25d484c8930c4042ab7b4b15fc3afa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d484c8930c4042ab7b4b15fc3afa05" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 46217 } } }
I20260925 10:07:27.918005 8938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:27.918025 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:27.918229 8948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:27.918826 8948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:27.920564 8948 catalog_manager.cc:1347] Generated new cluster ID: 7d44dc4d8b0d494298c798470f883797
I20260925 10:07:27.920621 8948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:27.923481 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:27.917472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42487 (local address 127.19.46.254:46217)
0925 10:07:27.917908 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:27.917919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:27.917963 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:07:27.918091 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:27.918098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:27.918163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:27.918610 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:27.918763 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:27.920258 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:27.921272 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:27.921815 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:27.921966 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:27.922764 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:27.922768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:27.922768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:27.922967 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:27.923003 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:27.923015 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:27.923119 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:27.923206 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:07:27.923239 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":91,"threads_started":1}
I20260925 10:07:27.924484 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:27.926759 8948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:27.927557 8948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:27.933092 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:27.926677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50227 (local address 127.19.46.254:46217)
0925 10:07:27.926714 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:27.926716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:27.929770 (+ 3054us) server_negotiation.cc:407] Connection header received
0925 10:07:27.929853 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:27.929854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:27.929866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:27.929941 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:27.930120 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:27.930682 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:27.931518 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:27.931767 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:27.931852 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:27.932800 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:27.932802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:27.932803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:27.932809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:27.932827 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:27.932832 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:27.932894 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:27.932961 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:07:27.932977 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:07:27.935746 8948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d484c8930c4042ab7b4b15fc3afa05: Generated new TSK 0
I20260925 10:07:27.935957 8948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:27.938205 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8866
W20260925 10:07:27.942508 8851 connection.cc:535] client connection to 127.19.46.254:46217 recv error: Network error: failed to read from TLS socket (remote: 127.19.46.254:46217): Connection reset by peer (error 104)
2026-09-25T10:07:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:27 dist-test-slave-f9rl krb5kdc[8965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:28 dist-test-slave-f9rl krb5kdc[8965](info): set up 2 sockets
Sep 25 10:07:28 dist-test-slave-f9rl krb5kdc[8965](info): commencing operation
krb5kdc: starting...
W20260925 10:07:29.031138 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:29 dist-test-slave-f9rl krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:29.160132 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43021
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:29.250047 8978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:29.250275 8978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:29.250299 8978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:29.251441 8978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:29.251480 8978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:29.251495 8978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:29.251507 8978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:29.251520 8978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:29.252988 8978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:43021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:29.253196 8978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:29.253379 8978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:29.258436 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:29.258441 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:29.258697 8978 server_base.cc:1034] running on GCE node
W20260925 10:07:29.258721 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:29.259001 8978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:29.259310 8978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:29.260478 8978 hybrid_clock.cc:648] HybridClock initialized: now 1790330849260467 us; error 55 us; skew 500 ppm
Sep 25 10:07:29 dist-test-slave-f9rl krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:29.263458 8978 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:29.264015 8978 webserver.cc:460] Webserver started at http://127.19.46.254:37525/ using document root <none> and password file <none>
I20260925 10:07:29.264298 8978 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:29.264361 8978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:29.264532 8978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:29.265839 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "9ed201e0a297406ba56e8b514ab53a1e"
format_stamp: "Formatted at 2026-09-25 10:07:29 on dist-test-slave-f9rl"
server_key: "a799ef409247bf2db7494f39766e066d"
server_key_iv: "f469c3124e337c297ac85fce973f3223"
server_key_version: "encryptionkey@0"
I20260925 10:07:29.266244 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "9ed201e0a297406ba56e8b514ab53a1e"
format_stamp: "Formatted at 2026-09-25 10:07:29 on dist-test-slave-f9rl"
server_key: "a799ef409247bf2db7494f39766e066d"
server_key_iv: "f469c3124e337c297ac85fce973f3223"
server_key_version: "encryptionkey@0"
I20260925 10:07:29.267776 8978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:07:29.268635 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:29.268924 8978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:07:29.269049 8978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "9ed201e0a297406ba56e8b514ab53a1e"
format_stamp: "Formatted at 2026-09-25 10:07:29 on dist-test-slave-f9rl"
server_key: "a799ef409247bf2db7494f39766e066d"
server_key_iv: "f469c3124e337c297ac85fce973f3223"
server_key_version: "encryptionkey@0"
I20260925 10:07:29.269146 8978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:29.273039 8978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:29.275821 8978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:29.276018 8978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:29.280799 8978 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:43021
I20260925 10:07:29.280860 9046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:43021 every 8 connection(s)
I20260925 10:07:29.281236 8978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:29.282290 9047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:29.284631 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e: Bootstrap starting.
I20260925 10:07:29.285230 9047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:29.285620 9047 log.cc:824] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:29.285707 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 8978
I20260925 10:07:29.285830 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:29.286099 19643 external_mini_cluster.cc:1428] Setting key 8db3c56ab86d95079d6365135c442c47
I20260925 10:07:29.286490 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e: No bootstrap required, opened a new log
I20260925 10:07:29.288811 9047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } }
I20260925 10:07:29.289002 9047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:29.289045 9047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed201e0a297406ba56e8b514ab53a1e, State: Initialized, Role: FOLLOWER
I20260925 10:07:29.289383 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:29.286686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58437 (local address 127.19.46.254:43021)
0925 10:07:29.286895 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:29.286904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:29.286940 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:07:29.287916 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:29.287926 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:29.287961 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:29.288125 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:29.288850 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:29.288854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:29.288855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:29.288860 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:29.288904 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:29.288918 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:29.288993 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:29.289132 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:07:29.289174 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260925 10:07:29.289177 9047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } }
I20260925 10:07:29.289618 9047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:29.289670 9047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:29.289721 9047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:29.290410 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:29.290572 9047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } }
I20260925 10:07:29.290692 9047 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed201e0a297406ba56e8b514ab53a1e; no voters:
I20260925 10:07:29.290860 9047 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:29.290917 9052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:29.291087 9052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [term 1 LEADER]: Becoming Leader. State: Replica: 9ed201e0a297406ba56e8b514ab53a1e, State: Running, Role: LEADER
I20260925 10:07:29.291127 9047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:29.291263 9052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } }
I20260925 10:07:29.291648 9052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed201e0a297406ba56e8b514ab53a1e. Latest consensus state: current_term: 1 leader_uuid: "9ed201e0a297406ba56e8b514ab53a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } } }
I20260925 10:07:29.291744 9052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:29.291666 9053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed201e0a297406ba56e8b514ab53a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed201e0a297406ba56e8b514ab53a1e" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43021 } } }
I20260925 10:07:29.291790 9053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:29.292011 9064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:29.292524 9064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:29.294471 9064 catalog_manager.cc:1347] Generated new cluster ID: 6937eed2d39f4715ae7cfcab3a390916
I20260925 10:07:29.294519 9064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:29.294883 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:29.293312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50713 (local address 127.19.46.254:43021)
0925 10:07:29.293349 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:29.293352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:29.293690 (+ 338us) server_negotiation.cc:407] Connection header received
0925 10:07:29.293811 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:29.293812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:29.293817 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:29.293866 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:29.294664 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:29.294665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:29.294666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:29.294670 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:29.294685 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:29.294689 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:29.294745 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:29.294803 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:07:29.294818 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:07:29.314996 9064 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:29.315230 9064 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:29.322253 9064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ed201e0a297406ba56e8b514ab53a1e: Generated new TSK 0
I20260925 10:07:29.322508 9064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:29.333066 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 8978
2026-09-25T10:07:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:29 dist-test-slave-f9rl krb5kdc[9078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:29 dist-test-slave-f9rl krb5kdc[9078](info): set up 2 sockets
Sep 25 10:07:29 dist-test-slave-f9rl krb5kdc[9078](info): commencing operation
krb5kdc: starting...
W20260925 10:07:30.419571 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:30.548877 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39617
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:30.633284 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:30.633554 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:30.633582 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:30.634765 9091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:30.634812 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:30.634827 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:30.634840 9091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:30.634858 9091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:30.636318 9091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:30.636554 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:30.636735 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:30.642350 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:30.642289 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:30.642275 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:30.642496 9091 server_base.cc:1034] running on GCE node
I20260925 10:07:30.642793 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:30.643157 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:30.644356 9091 hybrid_clock.cc:648] HybridClock initialized: now 1790330850644332 us; error 42 us; skew 500 ppm
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:30.647406 9091 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:30.647964 9091 webserver.cc:460] Webserver started at http://127.19.46.254:34185/ using document root <none> and password file <none>
I20260925 10:07:30.648186 9091 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:30.648234 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:30.648336 9091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:30.649595 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "60dd2cacb7cd4473a1fbf52767703a15"
format_stamp: "Formatted at 2026-09-25 10:07:30 on dist-test-slave-f9rl"
server_key: "e3232eaad5412748a7f613a20e8d5843"
server_key_iv: "d17adf201ecb0787258d046d9dbc2317"
server_key_version: "encryptionkey@0"
I20260925 10:07:30.649937 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "60dd2cacb7cd4473a1fbf52767703a15"
format_stamp: "Formatted at 2026-09-25 10:07:30 on dist-test-slave-f9rl"
server_key: "e3232eaad5412748a7f613a20e8d5843"
server_key_iv: "d17adf201ecb0787258d046d9dbc2317"
server_key_version: "encryptionkey@0"
I20260925 10:07:30.651484 9091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:30.652390 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:30.652640 9091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:30.652763 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "60dd2cacb7cd4473a1fbf52767703a15"
format_stamp: "Formatted at 2026-09-25 10:07:30 on dist-test-slave-f9rl"
server_key: "e3232eaad5412748a7f613a20e8d5843"
server_key_iv: "d17adf201ecb0787258d046d9dbc2317"
server_key_version: "encryptionkey@0"
I20260925 10:07:30.652851 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:30.663182 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:30.665913 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:30.666105 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:30.670709 9091 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:39617
I20260925 10:07:30.670743 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:39617 every 8 connection(s)
I20260925 10:07:30.671180 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:30.672190 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:30.674530 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9091
I20260925 10:07:30.674669 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:30.674876 19643 external_mini_cluster.cc:1428] Setting key c9090480ff6b0d628ddc398824a77269
I20260925 10:07:30.674882 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15: Bootstrap starting.
I20260925 10:07:30.675634 9159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:30.675963 9159 log.cc:824] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:30.676769 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15: No bootstrap required, opened a new log
I20260925 10:07:30.679159 9159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } }
I20260925 10:07:30.679313 9159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:30.679337 9159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dd2cacb7cd4473a1fbf52767703a15, State: Initialized, Role: FOLLOWER
I20260925 10:07:30.679476 9159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } }
I20260925 10:07:30.679549 9159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:30.679587 9159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:30.679636 9159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:30.680424 9159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } }
I20260925 10:07:30.680559 9159 leader_election.cc:302] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60dd2cacb7cd4473a1fbf52767703a15; no voters:
I20260925 10:07:30.680778 9159 leader_election.cc:288] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:30.680796 9164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:30.680940 9164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [term 1 LEADER]: Becoming Leader. State: Replica: 60dd2cacb7cd4473a1fbf52767703a15, State: Running, Role: LEADER
I20260925 10:07:30.681085 9164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } }
I20260925 10:07:30.681210 9159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:30.682622 9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60dd2cacb7cd4473a1fbf52767703a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } } }
I20260925 10:07:30.682797 9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [sys.catalog]: This master's current role is: LEADER
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:30.683277 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dd2cacb7cd4473a1fbf52767703a15. Latest consensus state: current_term: 1 leader_uuid: "60dd2cacb7cd4473a1fbf52767703a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dd2cacb7cd4473a1fbf52767703a15" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 39617 } } }
I20260925 10:07:30.683359 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:30.683502 9171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:30.684162 9171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:30.685823 9171 catalog_manager.cc:1347] Generated new cluster ID: f5d692fd0510481a866c8420d351656e
I20260925 10:07:30.686103 9171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:30.688102 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:30.675467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39255 (local address 127.19.46.254:39617)
0925 10:07:30.675745 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:30.675756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:30.675795 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:07:30.676380 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:30.676388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:30.676491 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:30.676642 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:30.677353 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:30.678808 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:30.679846 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:30.680496 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:30.680615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:30.684008 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:30.684013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:30.684013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:30.684025 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:30.686344 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:30.686762 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:30.686765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:30.686766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:30.686864 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:30.687108 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:30.687109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:30.687109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:30.687369 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:30.687582 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:30.687681 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:07:30.687851 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260925 10:07:30.689097 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:30.699795 9171 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:30.700681 9171 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:30.701294 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:30.693935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.19.46.254:39617)
0925 10:07:30.694013 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:30.694016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:30.694182 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:07:30.694272 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:30.694274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:30.694286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:30.694351 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:30.694979 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:30.695558 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:30.696339 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:30.696617 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:30.696707 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:30.698220 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:30.698221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:30.698222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:30.698233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:30.699969 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:30.700404 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:30.700405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:30.700406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:30.700471 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:30.700701 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:30.700701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:30.700702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:30.700834 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:30.700931 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:30.701105 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:07:30.701201 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:07:30.706236 9171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60dd2cacb7cd4473a1fbf52767703a15: Generated new TSK 0
I20260925 10:07:30.706465 9171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:30.711519 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9091
2026-09-25T10:07:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9191](info): set up 2 sockets
Sep 25 10:07:30 dist-test-slave-f9rl krb5kdc[9191](info): commencing operation
krb5kdc: starting...
W20260925 10:07:31.795204 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:31 dist-test-slave-f9rl krb5kdc[9191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:31.934084 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43969
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:32.018808 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:32.019052 9204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:32.019078 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:32.020408 9204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:32.020469 9204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:32.020485 9204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:32.020498 9204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:32.020510 9204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:32.022266 9204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:32.022569 9204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:32.022816 9204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:32.028095 9209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:32.028229 9204 server_base.cc:1034] running on GCE node
W20260925 10:07:32.028108 9212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:32.028360 9210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:32.028648 9204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:32.029001 9204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:32.030186 9204 hybrid_clock.cc:648] HybridClock initialized: now 1790330852030166 us; error 31 us; skew 500 ppm
Sep 25 10:07:32 dist-test-slave-f9rl krb5kdc[9191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330852, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:32.038726 9204 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:32.039348 9204 webserver.cc:460] Webserver started at http://127.19.46.254:38757/ using document root <none> and password file <none>
I20260925 10:07:32.039609 9204 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:32.039683 9204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:32.039847 9204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:32.041049 9204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "af0175034a00411f8fcb13a6d6987f5d"
format_stamp: "Formatted at 2026-09-25 10:07:32 on dist-test-slave-f9rl"
server_key: "fd4ceeedbb022f59d3615f06e494208c"
server_key_iv: "49f206aa18a02732644b5041ca711142"
server_key_version: "encryptionkey@0"
I20260925 10:07:32.041501 9204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "af0175034a00411f8fcb13a6d6987f5d"
format_stamp: "Formatted at 2026-09-25 10:07:32 on dist-test-slave-f9rl"
server_key: "fd4ceeedbb022f59d3615f06e494208c"
server_key_iv: "49f206aa18a02732644b5041ca711142"
server_key_version: "encryptionkey@0"
I20260925 10:07:32.043063 9204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:32.044037 9220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:32.044246 9204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:32.044361 9204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "af0175034a00411f8fcb13a6d6987f5d"
format_stamp: "Formatted at 2026-09-25 10:07:32 on dist-test-slave-f9rl"
server_key: "fd4ceeedbb022f59d3615f06e494208c"
server_key_iv: "49f206aa18a02732644b5041ca711142"
server_key_version: "encryptionkey@0"
I20260925 10:07:32.044524 9204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:32.056915 9204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:32.059576 9204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:32.059793 9204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:32.064653 9204 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:43969
I20260925 10:07:32.064718 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:43969 every 8 connection(s)
I20260925 10:07:32.065140 9204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:32.066226 9273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:32.068691 9273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d: Bootstrap starting.
I20260925 10:07:32.069288 9273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:32.069602 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9204
I20260925 10:07:32.069664 9273 log.cc:824] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:32.069736 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:32.070027 19643 external_mini_cluster.cc:1428] Setting key d766c4c791280573f94b752ccebe0aa6
I20260925 10:07:32.070636 9273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d: No bootstrap required, opened a new log
I20260925 10:07:32.074223 9273 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } }
I20260925 10:07:32.074417 9273 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:32.074469 9273 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0175034a00411f8fcb13a6d6987f5d, State: Initialized, Role: FOLLOWER
I20260925 10:07:32.074621 9273 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } }
I20260925 10:07:32.074730 9273 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:32.074774 9273 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:32.074832 9273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:32.075690 9273 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } }
I20260925 10:07:32.075819 9273 leader_election.cc:302] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0175034a00411f8fcb13a6d6987f5d; no voters:
I20260925 10:07:32.076061 9273 leader_election.cc:288] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:32.076130 9278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:32.076305 9278 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [term 1 LEADER]: Becoming Leader. State: Replica: af0175034a00411f8fcb13a6d6987f5d, State: Running, Role: LEADER
I20260925 10:07:32.076408 9278 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } }
I20260925 10:07:32.076879 9273 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:32.077191 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0175034a00411f8fcb13a6d6987f5d. Latest consensus state: current_term: 1 leader_uuid: "af0175034a00411f8fcb13a6d6987f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } } }
I20260925 10:07:32.077335 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:32.077806 9284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:32.078004 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:32.070746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56109 (local address 127.19.46.254:43969)
0925 10:07:32.070925 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:32.070936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:32.070974 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:07:32.071575 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:32.071585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:32.071658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:32.071845 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:32.072131 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:32.073823 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:32.075023 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:32.075705 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:32.075834 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:32.076769 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:32.076774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:32.076774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:32.077004 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:32.077047 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:32.077065 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:32.077151 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:32.077619 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 10:07:32.077703 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260925 10:07:32.076853 9280 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0175034a00411f8fcb13a6d6987f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0175034a00411f8fcb13a6d6987f5d" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 43969 } } }
I20260925 10:07:32.078378 9284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:32.078405 9280 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:32.079224 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:32.080403 9284 catalog_manager.cc:1347] Generated new cluster ID: f177e3c9673743b18c5c0450d574a961
I20260925 10:07:32.080469 9284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:32.089759 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:32.081887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44551 (local address 127.19.46.254:43969)
0925 10:07:32.081932 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:32.081934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:32.085729 (+ 3795us) server_negotiation.cc:407] Connection header received
0925 10:07:32.085811 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:32.085813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:32.085827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:32.085903 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:32.086152 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:32.086980 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:32.088156 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:32.088504 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:32.088579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:32.089465 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:32.089466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:32.089467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:32.089474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:32.089491 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:32.089496 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:32.089552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:32.089633 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 10:07:32.089663 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:07:32.100641 9284 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:32.101601 9284 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:32.106709 9284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0175034a00411f8fcb13a6d6987f5d: Generated new TSK 0
I20260925 10:07:32.106949 9284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:32.113173 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9204
2026-09-25T10:07:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:32 dist-test-slave-f9rl krb5kdc[9304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:32 dist-test-slave-f9rl krb5kdc[9304](info): set up 2 sockets
Sep 25 10:07:32 dist-test-slave-f9rl krb5kdc[9304](info): commencing operation
krb5kdc: starting...
W20260925 10:07:33.215093 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:33 dist-test-slave-f9rl krb5kdc[9304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:33.345014 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38561
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:33.427233 9318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:33.427459 9318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:33.427484 9318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:33.428627 9318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:33.428683 9318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:33.428699 9318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:33.428710 9318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:33.428723 9318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:33.430209 9318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:33.430425 9318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:33.430608 9318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:33.435830 9326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:33.435802 9324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:33.435817 9323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:33.436080 9318 server_base.cc:1034] running on GCE node
I20260925 10:07:33.436372 9318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:33.436671 9318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:33.437858 9318 hybrid_clock.cc:648] HybridClock initialized: now 1790330853437833 us; error 40 us; skew 500 ppm
Sep 25 10:07:33 dist-test-slave-f9rl krb5kdc[9304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330853, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:33.440673 9318 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:33.441175 9318 webserver.cc:460] Webserver started at http://127.19.46.254:42411/ using document root <none> and password file <none>
I20260925 10:07:33.441426 9318 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:33.441540 9318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:33.441702 9318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:33.442903 9318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "9e19589d24fb42e39b10bb459b5e8cf0"
format_stamp: "Formatted at 2026-09-25 10:07:33 on dist-test-slave-f9rl"
server_key: "fc6d4b3e9f3cfea176fe16f8b542045c"
server_key_iv: "b27d7cdb064752d5270f85106f1c6542"
server_key_version: "encryptionkey@0"
I20260925 10:07:33.443279 9318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "9e19589d24fb42e39b10bb459b5e8cf0"
format_stamp: "Formatted at 2026-09-25 10:07:33 on dist-test-slave-f9rl"
server_key: "fc6d4b3e9f3cfea176fe16f8b542045c"
server_key_iv: "b27d7cdb064752d5270f85106f1c6542"
server_key_version: "encryptionkey@0"
I20260925 10:07:33.444679 9318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:33.445614 9333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:33.445860 9318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:33.445945 9318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "9e19589d24fb42e39b10bb459b5e8cf0"
format_stamp: "Formatted at 2026-09-25 10:07:33 on dist-test-slave-f9rl"
server_key: "fc6d4b3e9f3cfea176fe16f8b542045c"
server_key_iv: "b27d7cdb064752d5270f85106f1c6542"
server_key_version: "encryptionkey@0"
I20260925 10:07:33.446045 9318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:33.449819 9318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:33.452527 9318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:33.452651 9318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:33.457204 9318 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:38561
I20260925 10:07:33.457238 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:38561 every 8 connection(s)
I20260925 10:07:33.457645 9318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:33.458873 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:33.460253 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9318
I20260925 10:07:33.460381 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:33.460567 19643 external_mini_cluster.cc:1428] Setting key d6476114b516d48b5cd43cd29f682e76
I20260925 10:07:33.462100 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0: Bootstrap starting.
I20260925 10:07:33.462769 9386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:33.463106 9386 log.cc:824] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:33.463547 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:33.461397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41775 (local address 127.19.46.254:38561)
0925 10:07:33.461704 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:33.461713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:33.461741 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:07:33.462178 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:33.462185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:33.462216 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:33.462387 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:33.463152 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:33.463156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:33.463156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:33.463161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:33.463205 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:33.463219 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:33.463296 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:33.463334 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 10:07:33.463374 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":150,"threads_started":1}
I20260925 10:07:33.464143 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0: No bootstrap required, opened a new log
I20260925 10:07:33.464814 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:33.466776 9386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } }
I20260925 10:07:33.466996 9386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:33.467043 9386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e19589d24fb42e39b10bb459b5e8cf0, State: Initialized, Role: FOLLOWER
I20260925 10:07:33.467180 9386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } }
I20260925 10:07:33.467303 9386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:33.467350 9386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:33.467419 9386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:33.467808 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:33.466333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47809 (local address 127.19.46.254:38561)
0925 10:07:33.466399 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:33.466401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:33.466460 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:07:33.466507 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:33.466508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:33.466513 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:33.466588 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:33.467279 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:33.467280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:33.467281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:07:33.467285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:33.467303 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:07:33.467308 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:33.467387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:33.467745 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:07:33.467759 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:07:33.468462 9386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } }
I20260925 10:07:33.468562 9386 leader_election.cc:302] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e19589d24fb42e39b10bb459b5e8cf0; no voters:
I20260925 10:07:33.468683 9386 leader_election.cc:288] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:33.468727 9396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:33.468900 9396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 9e19589d24fb42e39b10bb459b5e8cf0, State: Running, Role: LEADER
I20260925 10:07:33.468997 9386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:33.468989 9396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } }
I20260925 10:07:33.469398 9397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } } }
I20260925 10:07:33.469527 9397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:33.469699 9398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e19589d24fb42e39b10bb459b5e8cf0. Latest consensus state: current_term: 1 leader_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e19589d24fb42e39b10bb459b5e8cf0" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 38561 } } }
I20260925 10:07:33.469724 9402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:33.469776 9398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:33.470217 9402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:33.471939 9402 catalog_manager.cc:1347] Generated new cluster ID: c9e4eb100a2849da9d55c6a485506fd3
I20260925 10:07:33.472004 9402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:33.482051 9402 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:33.482471 9402 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:33.486327 9402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e19589d24fb42e39b10bb459b5e8cf0: Generated new TSK 0
I20260925 10:07:33.486611 9402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:33.492146 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9318
2026-09-25T10:07:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1378 ms)
[----------] 18 tests from AuthTokenIssuingTest (25805 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:33 dist-test-slave-f9rl krb5kdc[9417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:33 dist-test-slave-f9rl krb5kdc[9417](info): set up 2 sockets
Sep 25 10:07:33 dist-test-slave-f9rl krb5kdc[9417](info): commencing operation
krb5kdc: starting...
W20260925 10:07:34.612049 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:34 dist-test-slave-f9rl krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:34.744693 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40217
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:42719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:34.833151 9430 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:34.833501 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:34.833539 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:34.833585 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:34.835044 9430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:34.835109 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:34.835126 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:34.835141 9430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:34.835155 9430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:34.837087 9430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:42719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:34.837426 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:34.837769 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:34.843353 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:34.843556 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:34.843767 9430 server_base.cc:1034] running on GCE node
W20260925 10:07:34.843918 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:34.844241 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:34.844609 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:34.845815 9430 hybrid_clock.cc:648] HybridClock initialized: now 1790330854845797 us; error 32 us; skew 500 ppm
Sep 25 10:07:34 dist-test-slave-f9rl krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:34.848852 9430 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:34.849373 9430 webserver.cc:460] Webserver started at http://127.19.46.254:38051/ using document root <none> and password file <none>
I20260925 10:07:34.849707 9430 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:34.849761 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:34.849900 9430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:34.851024 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "02f5a1444a9f4aa4988c094d74ed7ef9"
format_stamp: "Formatted at 2026-09-25 10:07:34 on dist-test-slave-f9rl"
server_key: "cfe73ebd5aced31dcef072afa56b5934"
server_key_iv: "6521670df941f700068ad38e4fbe912d"
server_key_version: "encryptionkey@0"
I20260925 10:07:34.851404 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "02f5a1444a9f4aa4988c094d74ed7ef9"
format_stamp: "Formatted at 2026-09-25 10:07:34 on dist-test-slave-f9rl"
server_key: "cfe73ebd5aced31dcef072afa56b5934"
server_key_iv: "6521670df941f700068ad38e4fbe912d"
server_key_version: "encryptionkey@0"
I20260925 10:07:34.852931 9430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:34.853886 9446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:34.854141 9430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:34.854250 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "02f5a1444a9f4aa4988c094d74ed7ef9"
format_stamp: "Formatted at 2026-09-25 10:07:34 on dist-test-slave-f9rl"
server_key: "cfe73ebd5aced31dcef072afa56b5934"
server_key_iv: "6521670df941f700068ad38e4fbe912d"
server_key_version: "encryptionkey@0"
I20260925 10:07:34.854321 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:34.887068 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:34.889824 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:34.889977 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:34.894618 9430 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:40217
I20260925 10:07:34.894656 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:40217 every 8 connection(s)
I20260925 10:07:34.895197 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:34.896417 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:34.898176 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:34.900729 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9430
I20260925 10:07:34.900876 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:34.901041 19643 external_mini_cluster.cc:1428] Setting key e5cd149770e4f937e4da58858f41731e
Sep 25 10:07:34 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.253@KRBTEST.COM; defaulting to no policy
I20260925 10:07:34.911022 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.898981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40217 (local address 127.0.0.1:46108)
0925 10:07:34.899185 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:34.899238 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:34.899741 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:34.900087 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:34.900089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:34.900356 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:34.900598 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:34.900604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:34.901992 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:34.901993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:34.903145 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:34.903148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:34.903698 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:34.903699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:34.903764 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:34.904611 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:34.904621 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:34.906557 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:34.909091 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:34.909102 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:34.909105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:34.909362 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:34.909746 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:34.909747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:34.909747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:34.909901 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:34.910473 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:34.910480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:34.910565 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 10:07:34.910662 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:07:34.910828 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260925 10:07:34.911098 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.899063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.19.46.254:40217)
0925 10:07:34.899274 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:34.899281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:34.899320 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:07:34.899943 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:34.899945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:34.899963 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:34.900047 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:34.900711 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:34.901881 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:34.903243 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:34.903579 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:34.903715 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:34.906697 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:34.906706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:34.906706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:34.906714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:34.908956 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:34.909537 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:34.909540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:34.909541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:34.909650 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:34.910023 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:34.910024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:34.910025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:34.910335 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:34.910422 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:34.910740 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 10:07:34.910990 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260925 10:07:34.912083 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.253@KRBTEST.COM" created.
I20260925 10:07:34.912487 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.912351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:55980)
0925 10:07:34.912406 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:34.912462 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:07:34.912678 9450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
I20260925 10:07:34.913228 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.913094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:55982)
0925 10:07:34.913149 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:34.913181 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:07:34.913429 9499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:33271: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
I20260925 10:07:34.930030 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:33271
--webserver_interface=127.19.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:42719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:07:34.962728 9499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } attempt: 1
I20260925 10:07:34.963263 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.963099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:55998)
0925 10:07:34.963177 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:34.963229 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:07:34.963683 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:34.963548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56004)
0925 10:07:34.963608 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:34.963653 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:07:34.963908 9499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:33271: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
W20260925 10:07:35.012905 9506 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:35.013160 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:35.013180 9506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:35.013203 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:35.014531 9506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:35.014592 9506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:35.014608 9506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:35.014621 9506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:35.014634 9506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:35.016273 9506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:42719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:35.016583 9506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:35.016840 9506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:07:35.022070 9506 server_base.cc:1034] running on GCE node
W20260925 10:07:35.022066 9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:35.022063 9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:35.022192 9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:35.022596 9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:35.022976 9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:35.024212 9506 hybrid_clock.cc:648] HybridClock initialized: now 1790330855024167 us; error 59 us; skew 500 ppm
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:35.027351 9506 init.cc:377] Logged in from keytab as kudu/127.19.46.253@KRBTEST.COM (short username kudu)
I20260925 10:07:35.027908 9506 webserver.cc:460] Webserver started at http://127.19.46.253:35017/ using document root <none> and password file <none>
I20260925 10:07:35.028180 9506 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:35.028259 9506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:35.028440 9506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:35.029887 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data/instance:
uuid: "633fac37773a44cdb52c189ee2a97ff0"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "32b7b7909ebb4559a5a2fd1636cf4b3c"
server_key_iv: "a999ec3f9f168b3f11bede1a4379e679"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.030305 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal/instance:
uuid: "633fac37773a44cdb52c189ee2a97ff0"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "32b7b7909ebb4559a5a2fd1636cf4b3c"
server_key_iv: "a999ec3f9f168b3f11bede1a4379e679"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.031983 9506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:07:35.032164 9499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } attempt: 2
I20260925 10:07:35.032682 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.032502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56008)
0925 10:07:35.032575 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.032640 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 10:07:35.032990 9521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:35.033160 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.033042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56022)
0925 10:07:35.033099 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.033131 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:07:35.033174 9506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
W20260925 10:07:35.033259 9499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:33271: Network error: Client connection negotiation failed: client connection to 127.19.46.253:33271: connect: Connection refused (error 111)
I20260925 10:07:35.033304 9506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "633fac37773a44cdb52c189ee2a97ff0"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "32b7b7909ebb4559a5a2fd1636cf4b3c"
server_key_iv: "a999ec3f9f168b3f11bede1a4379e679"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.033594 9506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:35.053293 9506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:35.056154 9506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:35.056406 9506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:35.061232 9506 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.253:33271
I20260925 10:07:35.061270 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.253:33271 every 8 connection(s)
I20260925 10:07:35.061748 9506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
I20260925 10:07:35.062812 9574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:35.064524 9574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.066181 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9506
I20260925 10:07:35.066401 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-1/wal/instance
I20260925 10:07:35.066650 19643 external_mini_cluster.cc:1428] Setting key 189d9dbab4916f738f88d73c1ce56116
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.46.252@KRBTEST.COM; defaulting to no policy
I20260925 10:07:35.077747 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.065370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.19.46.254:40217)
0925 10:07:35.065685 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.065688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.066465 (+ 777us) server_negotiation.cc:407] Connection header received
0925 10:07:35.067316 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.067318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.067331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.067399 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.068554 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.069355 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.070666 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.070979 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.071219 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.074341 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.074343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.074344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.074356 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.076149 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.076632 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.076633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.076634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.076699 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.076971 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.076972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.076972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.077102 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.077180 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.077551 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.077645 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260925 10:07:35.077999 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.065260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40217 (local address 127.0.0.1:46120)
0925 10:07:35.065944 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.066208 (+ 264us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.067163 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.067514 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.067519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.068042 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.068394 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.068402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.069558 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.069559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.070551 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.070554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.071272 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.071274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.071469 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.072139 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.072152 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.074213 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.076273 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.076284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.076287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.076546 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.076772 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.076773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.076773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.076895 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.077173 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.077180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.077363 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.077532 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.077712 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":396,"threads_started":1}
I20260925 10:07:35.078869 9574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.252@KRBTEST.COM" created.
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:35.089092 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.079462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.19.46.253:33271)
0925 10:07:35.079650 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.079655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.079676 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:07:35.079696 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.079698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.079710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.079808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.080197 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.081231 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.082332 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.082808 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.082935 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.085341 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.085355 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.085355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.085364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.087312 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.087837 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.087841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.087841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.087956 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.088198 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.088199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.088200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.088438 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.088540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.088829 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.088951 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260925 10:07:35.090047 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.079286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56026)
0925 10:07:35.079350 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.079365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.079422 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.079807 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.079808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.079988 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.080094 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.080096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.081342 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.081343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.082231 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.082234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.082935 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.082936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.083008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.083589 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.083594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.085219 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.087535 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.087536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.087536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.087727 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.088034 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.088035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.088035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.088100 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.088548 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.088549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.088647 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.088793 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.089080 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":88}
I20260925 10:07:35.090866 9574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.091372 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.091194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40144)
0925 10:07:35.091265 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.091325 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:07:35.091548 9523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
I20260925 10:07:35.092197 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.092054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40146)
0925 10:07:35.092141 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.092174 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 10:07:35.092463 9574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:33147: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
I20260925 10:07:35.105247 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:33147
--webserver_interface=127.19.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:42719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:07:35.124439 9499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } attempt: 3
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:35.132673 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.125020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.19.46.253:33271)
0925 10:07:35.125106 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.125109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.125121 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:35.125153 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.125154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.125168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.125240 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.125696 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.126273 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.127117 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.127373 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.127460 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.129684 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.129686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.129686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.129695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.131342 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.131757 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.131758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.131759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.131828 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.132061 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.132062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.132063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.132197 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.132284 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.132463 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.132561 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:07:35.132673 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.124920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56036)
0925 10:07:35.124982 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.125008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.125083 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.125263 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.125265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.125516 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.125599 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.125601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.126383 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.126384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.127016 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.127018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.127472 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.127474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.127569 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.128097 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.128104 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.129575 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.131463 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.131464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.131465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.131670 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.131905 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.131906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.131907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.131976 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.132275 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.132276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.132330 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.132433 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.132555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:07:35.133356 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.133919 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.133727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40154)
0925 10:07:35.133847 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.133887 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 10:07:35.134753 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.134621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40170)
0925 10:07:35.134686 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.134717 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:07:35.134861 9499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:33147: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
I20260925 10:07:35.141726 9574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } attempt: 1
I20260925 10:07:35.142305 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.142123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40184)
0925 10:07:35.142216 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.142262 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:07:35.142776 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.142669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40188)
0925 10:07:35.142722 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.142753 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:07:35.142913 9574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:33147: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
I20260925 10:07:35.165967 9499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } attempt: 1
I20260925 10:07:35.167002 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.166841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40198)
0925 10:07:35.166919 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.166966 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:07:35.167512 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.167371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40208)
0925 10:07:35.167446 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.167475 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:07:35.167649 9499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:33147: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
W20260925 10:07:35.198654 9582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:35.198895 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:35.198913 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:35.198935 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:35.200016 9582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:35.200052 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:35.200065 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:35.200078 9582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:35.200089 9582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:35.201642 9582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:42719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:40217,127.19.46.253:33271,127.19.46.252:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:35.201896 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:35.202080 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:35.207237 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:35.207245 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:35.207245 9587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:35.207775 9582 server_base.cc:1034] running on GCE node
I20260925 10:07:35.208024 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:35.208309 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:35.209654 9582 hybrid_clock.cc:648] HybridClock initialized: now 1790330855209633 us; error 43 us; skew 500 ppm
I20260925 10:07:35.211068 9574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } attempt: 2
I20260925 10:07:35.211504 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.211359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40218)
0925 10:07:35.211431 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.211475 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:07:35.211983 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.211893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40220)
0925 10:07:35.211944 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.211968 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:07:35.212097 9574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:33147: Network error: Client connection negotiation failed: client connection to 127.19.46.252:33147: connect: Connection refused (error 111)
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:35.213186 9582 init.cc:377] Logged in from keytab as kudu/127.19.46.252@KRBTEST.COM (short username kudu)
I20260925 10:07:35.213759 9582 webserver.cc:460] Webserver started at http://127.19.46.252:40585/ using document root <none> and password file <none>
I20260925 10:07:35.214047 9582 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:35.214103 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:35.214255 9582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:35.215458 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data/instance:
uuid: "1eb52bbc19e0400bb25d01eab49938a4"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "ffd854e63c90693cc9670781777db80c"
server_key_iv: "bb9d88302d93689c1f291a28f648a0a5"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.215871 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal/instance:
uuid: "1eb52bbc19e0400bb25d01eab49938a4"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "ffd854e63c90693cc9670781777db80c"
server_key_iv: "bb9d88302d93689c1f291a28f648a0a5"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.217536 9582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 10:07:35.218731 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:35.219096 9582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:35.219184 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "1eb52bbc19e0400bb25d01eab49938a4"
format_stamp: "Formatted at 2026-09-25 10:07:35 on dist-test-slave-f9rl"
server_key: "ffd854e63c90693cc9670781777db80c"
server_key_iv: "bb9d88302d93689c1f291a28f648a0a5"
server_key_version: "encryptionkey@0"
I20260925 10:07:35.219297 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:35.231189 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:35.233290 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:35.233417 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:35.237668 9582 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.252:33147
I20260925 10:07:35.237783 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.252:33147 every 8 connection(s)
I20260925 10:07:35.238099 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
I20260925 10:07:35.239142 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:35.240789 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.241673 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9582
I20260925 10:07:35.241797 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330556283750-19643-0/minicluster-data/master-2/wal/instance
I20260925 10:07:35.241994 19643 external_mini_cluster.cc:1428] Setting key d5f27ecc16ba4316e34d2dab5d579226
I20260925 10:07:35.242794 9499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } attempt: 2
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:07:35.252681 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.241642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.19.46.254:40217)
0925 10:07:35.241688 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.241690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.242100 (+ 410us) server_negotiation.cc:407] Connection header received
0925 10:07:35.242670 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.242672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.242685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.242716 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.243597 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.244315 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.245588 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.245871 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.245961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.248454 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.248456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.248456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.248463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.250568 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.251230 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.251231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.251232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.251305 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.251729 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.251730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.251731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.252002 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.252055 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.252474 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.252560 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":18}
I20260925 10:07:35.252982 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.243066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40242)
0925 10:07:35.243117 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.243129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.243170 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.243444 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.243445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.243614 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.243664 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.243665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.245014 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.245015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.245839 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.245842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.246461 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.246463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.246514 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.247014 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.247019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.248915 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.251556 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.251557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.251557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.251735 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.252196 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.252196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.252197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.252261 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.252632 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.252633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.252686 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.252769 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.252891 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260925 10:07:35.253676 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.241655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:40217 (local address 127.0.0.1:46122)
0925 10:07:35.241898 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.241966 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.242535 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.242867 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.242870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.243275 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.243492 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.243498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.244405 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.244406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.245483 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.245486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.246062 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.246063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.246201 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.246869 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.246879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.248370 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.250681 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.250689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.250692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.251166 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.251415 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.251416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.251416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.251613 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.252117 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.252125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.252315 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.252445 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.253139 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":30,"thread_start_us":124,"threads_started":1}
I20260925 10:07:35.253711 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.243100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.19.46.252:33147)
0925 10:07:35.243294 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.243296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.243304 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:07:35.243324 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.243325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.243335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.243411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.243751 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.244920 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.245964 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.246293 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.246354 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.249008 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.249016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.249016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.249023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.251449 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.252052 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.252056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.252056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.252125 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.252328 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.252329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.252329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.252533 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.252664 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.253555 (+ 891us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.253620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260925 10:07:35.254174 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.242545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.19.46.252:33147)
0925 10:07:35.242707 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.242713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.242830 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:07:35.242868 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.242871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.242896 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.242986 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.243997 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.244932 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.246596 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.246942 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.247009 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.250872 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.250873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.250873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.250881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.252805 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.253204 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.253205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.253206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.253266 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.253475 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.253476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.253476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.253618 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.253788 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.253909 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.254000 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260925 10:07:35.254308 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.256354 9499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: Bootstrap starting.
I20260925 10:07:35.257119 9499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:35.257537 9499 log.cc:824] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:35.258562 9499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: No bootstrap required, opened a new log
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:35.260452 9499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.260608 9499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f5a1444a9f4aa4988c094d74ed7ef9, State: Initialized, Role: FOLLOWER
I20260925 10:07:35.260692 9499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.261062 9499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:35.263053 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.254746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.19.46.253:33271)
0925 10:07:35.254794 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.254796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.254808 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:07:35.254848 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.254849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.254861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.254926 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.255246 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.255709 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.256849 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.257130 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.257222 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.259641 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.259643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.259643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.259651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.261684 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.262100 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.262102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.262102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.262163 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.262407 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.262408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.262409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.262558 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.262604 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.262837 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.262936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 10:07:35.263370 9668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:35.263430 9668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:35.263667 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.254672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:33271 (local address 127.0.0.1:56042)
0925 10:07:35.254736 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.254746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.254785 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.254934 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.254935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.255099 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.255161 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.255163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.255839 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.255840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.256731 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.256735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.257253 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.257254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.257347 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.257868 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.257874 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.259529 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.261759 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.261760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.261760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.262019 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.262247 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.262248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.262248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.262315 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.262656 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.262656 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.262713 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.263368 (+ 655us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.263564 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:07:35.264132 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:35.265590 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.266546 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.264267 9430 master_runner.cc:186] Error getting master registration for 127.19.46.253:33271: OK, instance_id { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" instance_seqno: 1790330855059921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:07:35.267436 9430 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:07:35.268019 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.257547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.19.46.254:40217)
0925 10:07:35.257628 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.257630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.257640 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:35.257661 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.257662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.257672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.257740 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.258526 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.259346 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.260473 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.260822 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.260901 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.264479 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.264480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.264481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.264489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.266355 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.266950 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.266951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.266951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.267018 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.267260 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.267261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.267261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.267426 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.267566 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.267827 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.267917 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:35.277295 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.267560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.19.46.252:33147)
0925 10:07:35.267614 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.267617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.267627 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:35.267697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.267698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.267707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.267760 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.268035 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.268861 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.269771 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.270104 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.270179 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.273509 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.273511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.273511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.273519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.275070 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.275415 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.275416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.275416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.275468 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.276652 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.276653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.276654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.276859 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.276931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.277098 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.277186 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:07:35.277348 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.264410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40254)
0925 10:07:35.267562 (+ 3152us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.267574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.267621 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.267761 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.267761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.267922 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.267955 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.267957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.268947 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.268948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.269689 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.269692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.270192 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.270193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.270264 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.271718 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.271724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.273319 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.275154 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.275155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.275156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.275323 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.275533 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.275535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.275536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.275636 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.276966 (+ 1330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.276967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.276996 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.277075 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.277236 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3134,"mutex_wait_us":73}
I20260925 10:07:35.278347 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.268988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.19.46.253:33271)
0925 10:07:35.269035 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.269038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.269085 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:07:35.269157 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.269157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.269169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.269227 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.269902 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.270623 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.271683 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.271876 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.271989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.274657 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.274658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.274659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.274667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.276494 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.276903 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.276904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.276905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.276984 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.277626 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.277627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.277628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.277769 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.277893 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.278165 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.278255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:07:35.278990 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: Bootstrap starting.
I20260925 10:07:35.279628 9650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:35.279927 9650 log.cc:824] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:35.280804 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: No bootstrap required, opened a new log
I20260925 10:07:35.281726 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:35.282656 9650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.282840 9650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb52bbc19e0400bb25d01eab49938a4, State: Initialized, Role: FOLLOWER
I20260925 10:07:35.282941 9650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.282946 9473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46124
I20260925 10:07:35.283087 9473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:07:35.283135 9473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:35.283272 9650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:35.283241 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.283344 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.284217 9473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.284518 9473 leader_election.cc:288] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [CANDIDATE]: Term 1 election: Requested vote from peers 633fac37773a44cdb52c189ee2a97ff0 (127.19.46.253:33271), 1eb52bbc19e0400bb25d01eab49938a4 (127.19.46.252:33147)
I20260925 10:07:35.285023 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1eb52bbc19e0400bb25d01eab49938a4"
I20260925 10:07:35.285130 9624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:35.286042 9624 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f5a1444a9f4aa4988c094d74ed7ef9 in term 1.
I20260925 10:07:35.286352 9448 leader_election.cc:302] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f5a1444a9f4aa4988c094d74ed7ef9, 1eb52bbc19e0400bb25d01eab49938a4; no voters:
I20260925 10:07:35.286531 9657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:35.286717 9657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 02f5a1444a9f4aa4988c094d74ed7ef9, State: Running, Role: LEADER
I20260925 10:07:35.286837 9657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.287307 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f5a1444a9f4aa4988c094d74ed7ef9. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.287390 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:35.289475 9582 master_runner.cc:186] Error getting master registration for 127.19.46.253:33271: OK, instance_id { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" instance_seqno: 1790330855059921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:07:35.291872 9582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:07:35.292065 9692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:35.292269 9692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 10:07:35.294688 9690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:35.294739 9690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:35.296393 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.286534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.19.46.253:33271)
0925 10:07:35.287519 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.287521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.287531 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:07:35.287556 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.287557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.287568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.287651 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.288348 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.289111 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.290212 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.290963 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.291035 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.292769 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.292771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.292771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.292779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.294951 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.295371 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.295372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.295373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.295431 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.295635 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.295635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.295636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.295782 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.295879 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.296200 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.296283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960}
I20260925 10:07:35.297843 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.285927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.19.46.254:40217)
0925 10:07:35.285974 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.285976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.286276 (+ 300us) server_negotiation.cc:407] Connection header received
0925 10:07:35.286383 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.286384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.286396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.286478 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.288450 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.289278 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.290456 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.290797 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.290843 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.294531 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.294533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.294533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.294541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.296394 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.296829 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.296830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.296831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.296891 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.297128 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.297129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.297130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.297265 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.297354 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.297648 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.297751 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:07:35.298832 9624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [term 1 FOLLOWER]: Refusing update from remote peer 02f5a1444a9f4aa4988c094d74ed7ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:35.299412 9689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:35.300794 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f5a1444a9f4aa4988c094d74ed7ef9. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.300909 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.301762 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.286268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.19.46.252:33147)
0925 10:07:35.286494 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.286496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.286888 (+ 392us) server_negotiation.cc:407] Connection header received
0925 10:07:35.287018 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.287020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.287031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.287088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.288514 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.289203 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.293285 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.293603 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.293672 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.297528 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.297530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.297530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.297539 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.299281 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.299727 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.299728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.299728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.299801 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.301164 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.301166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.301166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.301300 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.301408 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.301586 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.301665 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260925 10:07:35.303226 9574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } attempt: 3
Sep 25 10:07:35 dist-test-slave-f9rl krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:07:35.312786 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.303503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:33147 (local address 127.0.0.1:40268)
0925 10:07:35.303559 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:35.303572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:35.303625 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:35.303762 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:35.303763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:35.303909 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:35.303948 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.303950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.304877 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.304878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.305705 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:35.305707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.306169 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.306170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:35.306227 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.306696 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:35.306700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:35.309001 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:35.311351 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.311352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.311352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.311610 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.311871 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:35.311872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:35.311872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:35.311953 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:35.312316 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:35.312317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:35.312471 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 10:07:35.312539 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:07:35.312654 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 10:07:35.313113 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:35.303545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.19.46.252:33147)
0925 10:07:35.303580 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.303582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.303608 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:07:35.303679 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.303680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.303690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.303744 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.304079 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.304799 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.305779 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:35.306091 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:35.306160 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:35.309087 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:35.309088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:35.309089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:35.309096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:35.311234 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.311703 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.311705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.311705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.311785 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:35.312033 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:35.312034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:35.312034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:35.312215 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:35.312275 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:35.312946 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 10:07:35.313033 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:07:35.315577 9574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: Bootstrap starting.
I20260925 10:07:35.315763 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.315934 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:35.316025 9694 mvcc.cc:205] Tried to move back new op lower bound from 7333195183300599808 to 7333195183255822336. Current Snapshot: MvccSnapshot[applied={T|T < 7333195183300599808}]
I20260925 10:07:35.316679 9574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:35.316704 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.316802 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.317050 9574 log.cc:824] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:35.317196 9695 mvcc.cc:205] Tried to move back new op lower bound from 7333195183300599808 to 7333195183255822336. Current Snapshot: MvccSnapshot[applied={T|T < 7333195183300599808}]
I20260925 10:07:35.318017 9574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: No bootstrap required, opened a new log
I20260925 10:07:35.320537 9574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.320715 9574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633fac37773a44cdb52c189ee2a97ff0, State: Initialized, Role: FOLLOWER
I20260925 10:07:35.320807 9574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } }
I20260925 10:07:35.321183 9574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:35.321256 9547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "633fac37773a44cdb52c189ee2a97ff0"
I20260925 10:07:35.321552 9692 catalog_manager.cc:1347] Generated new cluster ID: 1e332cbfe48243f48335bdc0b87ae1dc
I20260925 10:07:35.321609 9692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:35.321300 9548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:35.322865 9548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [term 1 FOLLOWER]: Refusing update from remote peer 02f5a1444a9f4aa4988c094d74ed7ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:35.321172 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.322979 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.323863 9547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f5a1444a9f4aa4988c094d74ed7ef9 in term 1.
I20260925 10:07:35.325644 9657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:07:35.326953 9711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:35.327105 9711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:35.328379 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f5a1444a9f4aa4988c094d74ed7ef9. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.328449 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.328480 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.328518 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:35.330607 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f5a1444a9f4aa4988c094d74ed7ef9" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 40217 } } peers { permanent_uuid: "633fac37773a44cdb52c189ee2a97ff0" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 33271 } } peers { permanent_uuid: "1eb52bbc19e0400bb25d01eab49938a4" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 33147 } } }
I20260925 10:07:35.330710 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:35.332173 9692 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:35.332911 9692 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:35.349682 9692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02f5a1444a9f4aa4988c094d74ed7ef9: Generated new TSK 0
I20260925 10:07:35.349942 9692 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:07:35.379236 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.377531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.19.46.253:33271)
0925 10:07:35.377579 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.377581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.378507 (+ 926us) server_negotiation.cc:407] Connection header received
0925 10:07:35.378617 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.378619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.378632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.378664 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.379156 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:07:35.380115 9502 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.377960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.19.46.254:40217)
0925 10:07:35.378020 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.378022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.379559 (+ 1537us) server_negotiation.cc:407] Connection header received
0925 10:07:35.379595 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.379597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.379611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.379676 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.380039 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46140: BlockingRecv error: recv got EOF from 127.0.0.1:46140 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 10:07:35.380148 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.378226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.19.46.252:33147)
0925 10:07:35.378268 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.378270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.379636 (+ 1366us) server_negotiation.cc:407] Connection header received
0925 10:07:35.379732 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.379733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.379745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.379774 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.380089 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40276: BlockingRecv error: recv got EOF from 127.0.0.1:40276 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:07:35.383706 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.382318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.19.46.253:33271)
0925 10:07:35.382361 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.382363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.382645 (+ 282us) server_negotiation.cc:407] Connection header received
0925 10:07:35.383150 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.383152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.383174 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.383204 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.383649 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56092: BlockingRecv error: recv got EOF from 127.0.0.1:56092 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:07:35.387180 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.386247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.19.46.253:33271)
0925 10:07:35.386288 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.386290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.386579 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:07:35.386682 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.386683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.386694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.386723 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.387121 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56104: BlockingRecv error: recv got EOF from 127.0.0.1:56104 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:07:35.391755 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.390469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.19.46.253:33271)
0925 10:07:35.390521 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.390523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.390779 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:07:35.390882 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.390883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.390894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.390925 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.391692 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56108: BlockingRecv error: recv got EOF from 127.0.0.1:56108 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 10:07:35.398522 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.397405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.19.46.253:33271)
0925 10:07:35.397487 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.397490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.397809 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:07:35.397932 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.397933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.397946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.397979 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.398449 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56114: BlockingRecv error: recv got EOF from 127.0.0.1:56114 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:07:35.409947 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.408717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.19.46.253:33271)
0925 10:07:35.408771 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.408774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.409110 (+ 336us) server_negotiation.cc:407] Connection header received
0925 10:07:35.409253 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.409254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.409267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.409302 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.409851 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56122: BlockingRecv error: recv got EOF from 127.0.0.1:56122 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 10:07:35.428529 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.427510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.19.46.253:33271)
0925 10:07:35.427556 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.427558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.427881 (+ 323us) server_negotiation.cc:407] Connection header received
0925 10:07:35.427994 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.427995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.428008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.428039 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.428464 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56124: BlockingRecv error: recv got EOF from 127.0.0.1:56124 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:07:35.463312 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.462212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.19.46.253:33271)
0925 10:07:35.462258 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.462260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.462603 (+ 343us) server_negotiation.cc:407] Connection header received
0925 10:07:35.462719 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.462721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.462734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.462766 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.463249 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56134: BlockingRecv error: recv got EOF from 127.0.0.1:56134 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:07:35.530069 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.529018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.19.46.253:33271)
0925 10:07:35.529073 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.529076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.529220 (+ 144us) server_negotiation.cc:407] Connection header received
0925 10:07:35.529260 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.529262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.529278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.529348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.530007 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56148: BlockingRecv error: recv got EOF from 127.0.0.1:56148 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 10:07:35.660256 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.659553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.19.46.253:33271)
0925 10:07:35.659640 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.659642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.659653 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:35.659680 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.659681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.659692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.659762 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.660196 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56150: BlockingRecv error: recv got EOF from 127.0.0.1:56150 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:07:35.918527 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:35.917840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.19.46.253:33271)
0925 10:07:35.917894 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:35.917896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:35.917941 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:07:35.918021 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:35.918022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:35.918033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:35.918110 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:35.918477 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56152: BlockingRecv error: recv got EOF from 127.0.0.1:56152 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:07:36.295251 9690 catalog_manager.cc:1259] Loaded cluster ID: 1e332cbfe48243f48335bdc0b87ae1dc
I20260925 10:07:36.295365 9690 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: loading cluster ID for follower catalog manager: success
I20260925 10:07:36.296228 9690 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: acquiring CA information for follower catalog manager: success
I20260925 10:07:36.296586 9690 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1eb52bbc19e0400bb25d01eab49938a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:07:36.327677 9711 catalog_manager.cc:1259] Loaded cluster ID: 1e332cbfe48243f48335bdc0b87ae1dc
I20260925 10:07:36.327759 9711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: loading cluster ID for follower catalog manager: success
I20260925 10:07:36.328871 9711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: acquiring CA information for follower catalog manager: success
I20260925 10:07:36.329336 9711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 633fac37773a44cdb52c189ee2a97ff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:07:36.434875 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:36.431986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.19.46.253:33271)
0925 10:07:36.432112 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:36.432114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:36.432142 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:07:36.432245 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:36.432246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:36.432337 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:36.432436 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:36.432596 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.433194 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.434016 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.434233 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.434308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:36.434486 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:36.434616 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:36.434701 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:36.434743 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:07:36.434766 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 10:07:36.439934 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:36.437238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.19.46.252:33147)
0925 10:07:36.437369 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:36.437371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:36.437382 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:36.437542 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:36.437542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:36.437581 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:36.437658 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:36.437744 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.438254 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.439080 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.439355 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.439427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:36.439587 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:36.439700 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:36.439766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:36.439844 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:07:36.439865 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260925 10:07:36.445294 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:36.442077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.19.46.253:33271)
0925 10:07:36.442132 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:36.442135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:36.442330 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:07:36.442443 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:36.442444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:36.442454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:36.442517 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:36.442711 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.443483 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.444490 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.444708 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.444797 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:36.444988 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:36.445026 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:36.445080 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:36.445163 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:07:36.445190 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:07:36.445660 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:36.442376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.19.46.252:33147)
0925 10:07:36.442421 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:36.442423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:36.442809 (+ 386us) server_negotiation.cc:407] Connection header received
0925 10:07:36.442871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:36.442872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:36.442884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:36.442957 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:36.443092 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.443604 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.444779 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.445070 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.445140 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:36.445286 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:36.445317 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:36.445379 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:36.445557 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:07:36.445587 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:07:36.446076 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:36.442068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.19.46.254:40217)
0925 10:07:36.442199 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:36.442201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:36.442263 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:07:36.442375 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:36.442376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:36.442440 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:36.442532 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:36.442711 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.443478 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.444673 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:36.444961 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:36.445018 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:36.445224 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:36.445378 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:36.445929 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:36.445951 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 10:07:36.445983 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260925 10:07:36.447988 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9430
I20260925 10:07:36.453783 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9506
I20260925 10:07:36.458688 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9582
2026-09-25T10:07:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2968 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:36 dist-test-slave-f9rl krb5kdc[9800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:36 dist-test-slave-f9rl krb5kdc[9800](info): set up 2 sockets
Sep 25 10:07:36 dist-test-slave-f9rl krb5kdc[9800](info): commencing operation
krb5kdc: starting...
W20260925 10:07:37.539312 19643 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/test-user.keytab.
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
Entry for principal kudu/127.19.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab.
I20260925 10:07:37.669373 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37685
--webserver_interface=127.19.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:37.753265 9813 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:37.753675 9813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:37.753705 9813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:37.753729 9813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:37.754959 9813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:37.755007 9813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:37.755023 9813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:37.755036 9813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:37.755049 9813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:37.756532 9813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.254.keytab
--principal=kudu/127.19.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.254:37685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:37.756747 9813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:37.756932 9813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:37.762306 9819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:37.762307 9822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:37.762351 9820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:37.762506 9813 server_base.cc:1034] running on GCE node
I20260925 10:07:37.762826 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:37.763092 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:37.764238 9813 hybrid_clock.cc:648] HybridClock initialized: now 1790330857764218 us; error 40 us; skew 500 ppm
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:37.772907 9813 init.cc:377] Logged in from keytab as kudu/127.19.46.254@KRBTEST.COM (short username kudu)
I20260925 10:07:37.773496 9813 webserver.cc:460] Webserver started at http://127.19.46.254:37199/ using document root <none> and password file <none>
I20260925 10:07:37.773763 9813 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:37.773813 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:37.773955 9813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:37.775072 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data/instance:
uuid: "c03d802917324bf7a82ffe2248664fed"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "de5101db8ceb0694d30e4bfa5c23ad93"
server_key_iv: "484c5a3bbd441923014852dc0f429c2b"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.775434 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance:
uuid: "c03d802917324bf7a82ffe2248664fed"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "de5101db8ceb0694d30e4bfa5c23ad93"
server_key_iv: "484c5a3bbd441923014852dc0f429c2b"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.776829 9813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:37.777726 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:37.777987 9813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:37.778064 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal
uuid: "c03d802917324bf7a82ffe2248664fed"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "de5101db8ceb0694d30e4bfa5c23ad93"
server_key_iv: "484c5a3bbd441923014852dc0f429c2b"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.778159 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:37.788666 9813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:37.790953 9813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:37.791122 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:37.795711 9813 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.254:37685
I20260925 10:07:37.795761 9881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.254:37685 every 8 connection(s)
I20260925 10:07:37.796139 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:37.797183 9882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:37.798797 9882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:37.804723 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9813
I20260925 10:07:37.804842 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-0/wal/instance
I20260925 10:07:37.805043 19643 external_mini_cluster.cc:1428] Setting key f47b2bf1a6c12cbef92461d0760987b9
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:37.809756 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.799609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37685 (local address 127.0.0.1:44856)
0925 10:07:37.799879 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.799920 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:37.800409 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:37.800722 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:37.800725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:37.800990 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:37.801170 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.801175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.802242 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.802243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.802999 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.803002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.803507 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.803508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.803571 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.804087 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:37.804095 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:37.805769 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:37.808148 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.808155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.808157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.808363 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.808635 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.808636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.808636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.808717 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.809178 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:37.809183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:37.809247 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 10:07:37.809325 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:07:37.809572 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":176,"threads_started":1}
I20260925 10:07:37.809875 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.799698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.19.46.254:37685)
0925 10:07:37.799880 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:37.799888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:37.800023 (+ 135us) server_negotiation.cc:407] Connection header received
0925 10:07:37.800541 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:37.800542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:37.800559 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:37.800680 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:37.801279 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.802148 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.803081 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.803386 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.803479 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.805890 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:37.805897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:37.805898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:37.805905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:37.808051 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.808451 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.808454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.808455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.808564 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.808783 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.808784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.808785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.809101 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:37.809195 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:37.809398 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:07:37.809688 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260925 10:07:37.810602 9882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:37.811003 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.810867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46170)
0925 10:07:37.810920 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.810967 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:07:37.811139 9833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
I20260925 10:07:37.811535 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.811444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46184)
0925 10:07:37.811489 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.811514 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:07:37.811671 9882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:42891: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
Entry for principal kudu/127.19.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab.
I20260925 10:07:37.833778 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:42891
--webserver_interface=127.19.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:07:37.860960 9882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } attempt: 1
I20260925 10:07:37.861536 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.861353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46190)
0925 10:07:37.861428 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.861504 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:07:37.861891 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.861783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46206)
0925 10:07:37.861830 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.861855 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:07:37.861977 9882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:42891: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
W20260925 10:07:37.917740 9889 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:37.917977 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:37.917994 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:37.918017 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:37.919255 9889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:37.919301 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:37.919317 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:37.919332 9889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:37.919344 9889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:37.920940 9889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.253.keytab
--principal=kudu/127.19.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.253:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:37.921223 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:37.921520 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:37.926661 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:37.926661 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:37.926702 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:37.926962 9889 server_base.cc:1034] running on GCE node
I20260925 10:07:37.927173 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:37.927507 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:37.928684 9889 hybrid_clock.cc:648] HybridClock initialized: now 1790330857928661 us; error 37 us; skew 500 ppm
I20260925 10:07:37.930132 9882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } attempt: 2
I20260925 10:07:37.930537 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.930409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46212)
0925 10:07:37.930472 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.930507 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:07:37.930882 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.930778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46218)
0925 10:07:37.930830 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.930865 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:07:37.931011 9882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.253:42891: Network error: Client connection negotiation failed: client connection to 127.19.46.253:42891: connect: Connection refused (error 111)
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:37.931608 9889 init.cc:377] Logged in from keytab as kudu/127.19.46.253@KRBTEST.COM (short username kudu)
I20260925 10:07:37.932075 9889 webserver.cc:460] Webserver started at http://127.19.46.253:43633/ using document root <none> and password file <none>
I20260925 10:07:37.932322 9889 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:37.932374 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:37.932550 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:37.933729 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data/instance:
uuid: "2a456ab21a3a47f5a102aef804dbda95"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "1969e06e3549f478ddb68d1107adbc2c"
server_key_iv: "5ae3a0116edf8258397daf783eb0110a"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.934074 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal/instance:
uuid: "2a456ab21a3a47f5a102aef804dbda95"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "1969e06e3549f478ddb68d1107adbc2c"
server_key_iv: "5ae3a0116edf8258397daf783eb0110a"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.935581 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:37.936496 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:37.936717 9889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:37.936812 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal
uuid: "2a456ab21a3a47f5a102aef804dbda95"
format_stamp: "Formatted at 2026-09-25 10:07:37 on dist-test-slave-f9rl"
server_key: "1969e06e3549f478ddb68d1107adbc2c"
server_key_iv: "5ae3a0116edf8258397daf783eb0110a"
server_key_version: "encryptionkey@0"
I20260925 10:07:37.936895 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:37.954916 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:37.957008 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:37.957170 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:37.961894 9889 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.253:42891
I20260925 10:07:37.961931 9957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.253:42891 every 8 connection(s)
I20260925 10:07:37.962328 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/data/info.pb
I20260925 10:07:37.963390 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:37.964980 9958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:37.969503 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9889
I20260925 10:07:37.969632 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-1/wal/instance
I20260925 10:07:37.969870 19643 external_mini_cluster.cc:1428] Setting key 3343ca441f63de52f79ca73b2d879606
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:37.975339 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.965847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.19.46.254:37685)
0925 10:07:37.965945 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:37.965948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:37.966139 (+ 191us) server_negotiation.cc:407] Connection header received
0925 10:07:37.966698 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:37.966699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:37.966714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:37.966779 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:37.967423 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.967928 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.968971 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.969207 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.969289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.971843 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:37.971845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:37.971845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:37.971854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:37.973672 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.974194 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.974195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.974196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.974276 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.974582 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.974583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.974584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.974759 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:37.974840 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:37.975096 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:07:37.975197 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:07:37.975457 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.965759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37685 (local address 127.0.0.1:44864)
0925 10:07:37.965993 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.966051 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:37.966586 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:37.966845 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:37.966849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:37.967112 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:37.967309 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.967312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.968033 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.968034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.968876 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.968878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.969308 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.969309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.969516 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.970135 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:37.970144 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:37.971746 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:37.973814 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.973830 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.973833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.974108 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.974397 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.974398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.974398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.974508 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.974833 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:37.974838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:37.974962 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 10:07:37.975046 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:07:37.975176 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260925 10:07:37.976293 9958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.46.252@KRBTEST.COM" created.
Sep 25 10:07:37 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:37.985647 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.976731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.19.46.253:42891)
0925 10:07:37.976959 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:37.976963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:37.976980 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:07:37.976998 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:37.976999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:37.977009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:37.977093 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:37.977428 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.978160 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.978929 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.979233 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.979321 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.982149 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:37.982160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:37.982160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:37.982168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:37.984237 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.984625 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.984628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.984629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.984690 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:37.984897 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:37.984898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:37.984898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:37.985072 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:37.985169 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:37.985350 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:07:37.985510 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":155,"threads_started":1}
I20260925 10:07:37.985659 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.976691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46228)
0925 10:07:37.976749 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.976758 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:37.976798 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:37.977093 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:37.977094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:37.977240 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:37.977290 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.977291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.978283 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.978284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.978831 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:37.978833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:37.979318 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:37.979319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:37.979388 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:37.979826 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:37.979830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:37.982075 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:37.984353 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.984353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.984354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.984526 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.984758 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:37.984759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:37.984760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:37.984826 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:37.985145 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:37.985146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:37.985186 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:07:37.985299 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:07:37.985576 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":149}
I20260925 10:07:37.986590 9958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:37.986984 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.986859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48356)
0925 10:07:37.986912 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.986964 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:07:37.987120 9909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
I20260925 10:07:37.987545 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:37.987440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48360)
0925 10:07:37.987492 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:37.987516 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:07:37.987643 9958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:45281: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
Entry for principal kudu/127.19.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab.
I20260925 10:07:38.000730 19643 external_mini_cluster.cc:1326] Running /tmp/dist-test-task46RfQ8/build/release/bin/kudu
/tmp/dist-test-task46RfQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:45281
--webserver_interface=127.19.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.46.212:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:07:38.022166 9882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } attempt: 3
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:38.029685 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.022675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.19.46.253:42891)
0925 10:07:38.022747 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.022750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.022761 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:38.022816 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.022817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.022829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.022901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.023238 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.023737 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.024446 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.024676 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.024761 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.026874 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.026876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.026876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.026883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.028373 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.028813 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.028815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.028815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.028885 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.029106 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.029107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.029107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.029230 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.029287 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.029540 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.029616 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:07:38.029691 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.022604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46238)
0925 10:07:38.022663 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.022678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.022757 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.022907 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.022908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.023081 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.023151 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.023153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.023824 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.023825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.024351 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.024353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.024758 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.024760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.024843 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.025332 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.025336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.026750 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.028461 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.028462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.028463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.028698 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.028965 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.028966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.028966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.029035 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.029304 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.029305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.029359 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.029461 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.029575 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:07:38.030275 9882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:38.030663 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.030539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48366)
0925 10:07:38.030598 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.030632 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:07:38.031005 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.030917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48380)
0925 10:07:38.030960 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.030984 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:07:38.031121 9882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:45281: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
I20260925 10:07:38.036875 9958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } attempt: 1
I20260925 10:07:38.037499 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.037329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48394)
0925 10:07:38.037391 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.037464 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:07:38.037873 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.037773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48400)
0925 10:07:38.037823 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.037851 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:07:38.037997 9958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:45281: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
I20260925 10:07:38.062278 9882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } attempt: 1
I20260925 10:07:38.062810 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.062668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48404)
0925 10:07:38.062739 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.062780 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:07:38.063174 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.063070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48408)
0925 10:07:38.063115 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.063140 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:07:38.063266 9882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:45281: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
W20260925 10:07:38.087208 9965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:07:38.087451 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:38.087468 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:38.087491 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:38.088851 9965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:38.088912 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:38.088927 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:38.088940 9965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:38.088953 9965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:38.090611 9965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.46.212:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.46.254:37685,127.19.46.253:42891,127.19.46.252:45281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/krb5kdc/kudu_127.19.46.252.keytab
--principal=kudu/127.19.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.46.252:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:38.090907 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:38.091176 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:38.096345 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:38.096335 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:38.096316 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:38.096743 9965 server_base.cc:1034] running on GCE node
I20260925 10:07:38.097045 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:38.097388 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:38.098610 9965 hybrid_clock.cc:648] HybridClock initialized: now 1790330858098597 us; error 40 us; skew 500 ppm
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:38.101536 9965 init.cc:377] Logged in from keytab as kudu/127.19.46.252@KRBTEST.COM (short username kudu)
I20260925 10:07:38.102063 9965 webserver.cc:460] Webserver started at http://127.19.46.252:39927/ using document root <none> and password file <none>
I20260925 10:07:38.102336 9965 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:38.102411 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:38.102566 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:38.103688 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data/instance:
uuid: "a1d526f4570c40af9939cfc1b7f6c645"
format_stamp: "Formatted at 2026-09-25 10:07:38 on dist-test-slave-f9rl"
server_key: "1c04e0b9d7f98cf1f986e776feb17ba6"
server_key_iv: "4238fea3fed41e16c6673fefda273e26"
server_key_version: "encryptionkey@0"
I20260925 10:07:38.104080 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal/instance:
uuid: "a1d526f4570c40af9939cfc1b7f6c645"
format_stamp: "Formatted at 2026-09-25 10:07:38 on dist-test-slave-f9rl"
server_key: "1c04e0b9d7f98cf1f986e776feb17ba6"
server_key_iv: "4238fea3fed41e16c6673fefda273e26"
server_key_version: "encryptionkey@0"
I20260925 10:07:38.105727 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:38.106171 9958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } attempt: 2
I20260925 10:07:38.106621 9980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:38.106809 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.106624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48416)
0925 10:07:38.106729 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.106778 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 10:07:38.106822 9965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:38.106968 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data,/tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal
uuid: "a1d526f4570c40af9939cfc1b7f6c645"
format_stamp: "Formatted at 2026-09-25 10:07:38 on dist-test-slave-f9rl"
server_key: "1c04e0b9d7f98cf1f986e776feb17ba6"
server_key_iv: "4238fea3fed41e16c6673fefda273e26"
server_key_version: "encryptionkey@0"
I20260925 10:07:38.107064 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:38.107240 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.107152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48426)
0925 10:07:38.107193 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.107215 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 10:07:38.107345 9958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.46.252:45281: Network error: Client connection negotiation failed: client connection to 127.19.46.252:45281: connect: Connection refused (error 111)
I20260925 10:07:38.120652 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:38.122952 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:38.123070 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:38.128046 9965 rpc_server.cc:307] RPC server started. Bound to: 127.19.46.252:45281
I20260925 10:07:38.128132 10032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.46.252:45281 every 8 connection(s)
I20260925 10:07:38.128476 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/data/info.pb
I20260925 10:07:38.129634 10033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:38.131209 10033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:38.136235 19643 external_mini_cluster.cc:1388] Started /tmp/dist-test-task46RfQ8/build/release/bin/kudu as pid 9965
I20260925 10:07:38.136435 19643 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task46RfQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330556283750-19643-0/minicluster-data/master-2/wal/instance
I20260925 10:07:38.136667 19643 external_mini_cluster.cc:1428] Setting key 362eca93fdd3a6dbd3accd5cd49b518c
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:38.138413 9882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } attempt: 2
I20260925 10:07:38.142683 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.132032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.19.46.254:37685)
0925 10:07:38.132150 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.132155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.132344 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:07:38.132796 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.132797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.132809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.132886 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.133566 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.134096 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.135142 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.135331 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.135428 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.138165 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.138167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.138167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.138175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.140684 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.141522 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.141524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.141525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.141587 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.141904 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.141905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.141905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.142029 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.142108 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.142481 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.142574 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:07:38.142855 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.131937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.254:37685 (local address 127.0.0.1:44872)
0925 10:07:38.132169 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.132211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.132687 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.132952 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.132955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.133208 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.133405 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.133409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.134202 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.134203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.135044 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.135047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.135511 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.135513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.135637 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.136267 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.136276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.138079 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.140777 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.140785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.140787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.141130 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.141666 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.141667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.141667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.141821 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.142099 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.142106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.142280 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.142405 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.142578 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.254@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:07:38.144114 10033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:38.146802 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.137047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.19.46.252:45281)
0925 10:07:38.137224 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.137230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.137279 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:07:38.137363 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.137368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.137394 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.137518 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.138184 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.139371 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.140290 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.140810 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.140930 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.143601 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.143610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.143611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.143618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.145579 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.145923 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.145926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.145926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.145990 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.146211 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.146212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.146213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.146368 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.146529 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.146601 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.146682 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260925 10:07:38.147583 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.138650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48454)
0925 10:07:38.138727 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.138739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.138782 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.139031 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.139032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.139183 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.139225 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.139227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.140352 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.140353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.141102 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.141105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.141526 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.141527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.141584 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.142157 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.142161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.143918 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.146505 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.146506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.146507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.146726 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.146967 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.146968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.146968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.147032 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.147291 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.147292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.147340 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.147409 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.147485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:07:38.147830 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.138731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.19.46.252:45281)
0925 10:07:38.138897 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.138899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.138906 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:07:38.138926 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.138927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.138937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.139276 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.139335 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.140234 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.141194 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.141407 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.141520 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.144014 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.144016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.144016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.144022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.146408 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.146802 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.146803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.146803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.146889 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.147101 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.147102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.147102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.147213 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.147272 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.147493 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.147610 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:38.149999 9882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: Bootstrap starting.
I20260925 10:07:38.151278 9882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:38.151919 9882 log.cc:824] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:38.152603 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.144866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.253:42891 (local address 127.0.0.1:46240)
0925 10:07:38.145149 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.145159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.145211 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.145373 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.145375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.145623 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.145676 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.145678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.146641 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.146642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.147498 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.147500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.147954 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.147955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.148017 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.148432 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.148437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.149832 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.151582 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.151583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.151583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.151750 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.151968 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.151968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.151969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.152031 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.152330 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.152331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.152390 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.152442 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.152535 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260925 10:07:38.152860 9882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: No bootstrap required, opened a new log
I20260925 10:07:38.152956 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.144942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.19.46.253:42891)
0925 10:07:38.145026 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.145028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.145218 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:07:38.145274 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.145275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.145288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.145357 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.145786 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.146557 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.147580 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.147877 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.147941 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.149939 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.149940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.149941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.149947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.151504 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.151835 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.151836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.151837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.151896 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.152086 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.152087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.152087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.152242 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.152734 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.152782 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.152856 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.254@KRBTEST.COM
I20260925 10:07:38.153738 10033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:07:38.155187 9882 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.155308 9882 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03d802917324bf7a82ffe2248664fed, State: Initialized, Role: FOLLOWER
I20260925 10:07:38.155416 9882 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.155753 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.155808 9882 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:38.155843 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.156924 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.148051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.19.46.254:37685)
0925 10:07:38.148092 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.148094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.148522 (+ 428us) server_negotiation.cc:407] Connection header received
0925 10:07:38.148594 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.148595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.148605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.148648 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.149205 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.149902 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.150899 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.151215 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.151290 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.153872 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.153874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.153874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.153886 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.155481 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.155943 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.155944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.155945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.156002 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.156217 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.156218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.156218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.156343 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.156506 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.156726 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.156810 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:07:38.157960 10052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:38.158010 10052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.252@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:07:38.162246 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.154034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48468)
0925 10:07:38.154136 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.154146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.154184 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.154480 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.154481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.154627 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.154665 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.154667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.155730 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.155731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.156451 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.156453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.157184 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.157185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.157241 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.157731 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.157736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.159264 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.161042 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.161043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.161043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.161316 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.161589 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.161590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.161590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.161647 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.161975 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.161976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.162012 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.162055 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.162148 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 10:07:38.162849 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.154094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.19.46.252:45281)
0925 10:07:38.154363 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.154365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.154373 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:07:38.154392 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.154393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.154404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.154464 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.154773 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.155632 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.156575 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.156905 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.157383 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.159357 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.159359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.159359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.159368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.160961 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.161402 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.161403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.161404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.161504 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.161710 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.161710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.161710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.161899 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.162638 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.162683 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.162767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.46.253@KRBTEST.COM
I20260925 10:07:38.159958 9813 master_runner.cc:186] Error getting master registration for 127.19.46.253:42891: OK, instance_id { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" instance_seqno: 1790330857960685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:07:38.163956 9813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:07:38.164340 10033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: Bootstrap starting.
I20260925 10:07:38.165047 10033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:38.165346 10033 log.cc:824] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:38.166420 10033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: No bootstrap required, opened a new log
I20260925 10:07:38.166777 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.157625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.19.46.253:42891)
0925 10:07:38.157668 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.157670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.157679 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:38.157732 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.157734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.157746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.157797 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.158414 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.159145 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.160189 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.160500 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.160561 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.163635 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.163636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.163637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.163644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.165257 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.165886 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.165887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.165887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.165955 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.166188 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.166189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.166190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.166320 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.166488 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.166591 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.166676 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:07:38.168005 19643 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:07:38.168469 10033 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.168627 10033 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d526f4570c40af9939cfc1b7f6c645, State: Initialized, Role: FOLLOWER
I20260925 10:07:38.168713 10033 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.168856 9856 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c03d802917324bf7a82ffe2248664fed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44874
I20260925 10:07:38.169020 9856 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:07:38.169040 10033 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:38.169003 10055 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.169067 9856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:38.169101 10055 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.170256 9856 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.170519 9856 leader_election.cc:288] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [CANDIDATE]: Term 1 election: Requested vote from peers 2a456ab21a3a47f5a102aef804dbda95 (127.19.46.253:42891), a1d526f4570c40af9939cfc1b7f6c645 (127.19.46.252:45281)
I20260925 10:07:38.171232 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c03d802917324bf7a82ffe2248664fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a1d526f4570c40af9939cfc1b7f6c645"
I20260925 10:07:38.171370 10007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:07:38.171921 10066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:38.171965 10066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:38.172339 10007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03d802917324bf7a82ffe2248664fed in term 1.
I20260925 10:07:38.172550 9833 leader_election.cc:302] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d526f4570c40af9939cfc1b7f6c645, c03d802917324bf7a82ffe2248664fed; no voters:
I20260925 10:07:38.172755 10041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:38.172910 10041 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [term 1 LEADER]: Becoming Leader. State: Replica: c03d802917324bf7a82ffe2248664fed, State: Running, Role: LEADER
I20260925 10:07:38.173010 10041 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.173516 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03d802917324bf7a82ffe2248664fed. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.173607 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:38.175426 10077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:38.176164 10077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:38.175289 9965 master_runner.cc:186] Error getting master registration for 127.19.46.253:42891: OK, instance_id { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" instance_seqno: 1790330857960685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:07:38.177946 9965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:07:38.186429 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.172202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.19.46.253:42891)
0925 10:07:38.172243 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.172246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.173166 (+ 920us) server_negotiation.cc:407] Connection header received
0925 10:07:38.173273 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.173274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.173285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.173317 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.174297 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.174995 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.179848 (+ 4853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.180200 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.180250 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.182454 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.182456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.182457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.182476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.184577 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.185198 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.185211 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.185211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.185273 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.185644 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.185645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.185645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.185792 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.185891 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.186207 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.186290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:07:38.188381 10007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [term 1 FOLLOWER]: Refusing update from remote peer c03d802917324bf7a82ffe2248664fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:38.188699 10041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:38.190029 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.172868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.19.46.254:37685)
0925 10:07:38.172921 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.172922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.178203 (+ 5281us) server_negotiation.cc:407] Connection header received
0925 10:07:38.178438 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.178440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.178452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.178486 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.180850 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.181593 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.183418 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.183718 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.183760 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.186453 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.186454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.186455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.186463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.188209 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.189083 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.189084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.189085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.189146 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.189387 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.189388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.189389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.189555 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.189629 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.189848 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.189942 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:07:38.190091 10055 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03d802917324bf7a82ffe2248664fed. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.190192 10055 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.190304 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.172718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.19.46.252:45281)
0925 10:07:38.174913 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.174915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.174922 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:07:38.174963 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.174964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.174973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.175046 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.178824 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.179565 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.183404 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.183718 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.183775 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.186141 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.186142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.186143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.186151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.188488 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.189080 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.189081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.189093 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.189146 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.189480 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.189482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.189482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.189610 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.189686 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.189995 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.190071 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177}
I20260925 10:07:38.194571 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.194684 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:38.194720 10055 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.194813 10055 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.194784 10078 mvcc.cc:205] Tried to move back new op lower bound from 7333195195134521344 to 7333195195077496832. Current Snapshot: MvccSnapshot[applied={T|T < 7333195195134521344}]
I20260925 10:07:38.195222 10077 catalog_manager.cc:1347] Generated new cluster ID: 224617c3aff14d53be2a38914b7625d3
I20260925 10:07:38.195268 10077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:07:38.198473 9958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } attempt: 3
Sep 25 10:07:38 dist-test-slave-f9rl krb5kdc[9800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.46.253@KRBTEST.COM for kudu/127.19.46.252@KRBTEST.COM
I20260925 10:07:38.207351 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.198734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.46.252:45281 (local address 127.0.0.1:48496)
0925 10:07:38.198814 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:38.198826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:38.198861 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:38.199158 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:38.199158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:38.199358 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:38.199407 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.199408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.200250 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.200251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.201041 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:38.201042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.202348 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.202349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:38.202423 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.202905 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:38.202923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:38.204452 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:38.206238 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.206239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.206240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.206463 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.206679 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:38.206679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:38.206680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:38.206744 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:38.207030 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:38.207031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:38.207087 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 10:07:38.207140 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:07:38.207237 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 10:07:38.207865 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:38.198924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.19.46.252:45281)
0925 10:07:38.198959 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.198961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.198967 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:07:38.198992 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.198993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.199002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.199042 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.199480 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.200171 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.201850 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:38.202136 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:38.202190 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:38.204554 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:38.204556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:38.204556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:38.204563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:38.206138 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.206544 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.206545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.206546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.206611 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:38.206812 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:38.206813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:38.206813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:38.206946 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:38.206995 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:38.207706 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 10:07:38.207781 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:07:38.209663 9958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: Bootstrap starting.
I20260925 10:07:38.210399 9958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:38.210666 9958 log.cc:824] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:38.212423 9958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: No bootstrap required, opened a new log
I20260925 10:07:38.214313 9958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.214454 9958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a456ab21a3a47f5a102aef804dbda95, State: Initialized, Role: FOLLOWER
I20260925 10:07:38.214525 9958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } }
I20260925 10:07:38.214854 9958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:38.216173 9931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:38.217191 9931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [term 1 FOLLOWER]: Refusing update from remote peer c03d802917324bf7a82ffe2248664fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:07:38.217207 10077 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:38.217314 10084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.217391 10084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.217749 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c03d802917324bf7a82ffe2248664fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a456ab21a3a47f5a102aef804dbda95"
I20260925 10:07:38.218056 10077 catalog_manager.cc:1504] Loading token signing keys...
W20260925 10:07:38.218240 10095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:07:38.218283 10095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:07:38.218675 9932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03d802917324bf7a82ffe2248664fed in term 1.
I20260925 10:07:38.219022 10074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:07:38.220548 10084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03d802917324bf7a82ffe2248664fed. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.220633 10084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.220670 10084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.220705 10084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:07:38.222365 10077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed: Generated new TSK 0
I20260925 10:07:38.222543 10077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:07:38.222760 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c03d802917324bf7a82ffe2248664fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03d802917324bf7a82ffe2248664fed" member_type: VOTER last_known_addr { host: "127.19.46.254" port: 37685 } } peers { permanent_uuid: "2a456ab21a3a47f5a102aef804dbda95" member_type: VOTER last_known_addr { host: "127.19.46.253" port: 42891 } } peers { permanent_uuid: "a1d526f4570c40af9939cfc1b7f6c645" member_type: VOTER last_known_addr { host: "127.19.46.252" port: 45281 } } }
I20260925 10:07:38.222843 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d802917324bf7a82ffe2248664fed [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:38.223815 10097 mvcc.cc:205] Tried to move back new op lower bound from 7333195195251130368 to 7333195195077496832. Current Snapshot: MvccSnapshot[applied={T|T < 7333195195134521344}]
W20260925 10:07:38.258327 9885 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.256051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.19.46.254:37685)
0925 10:07:38.256107 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.256110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.257302 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 10:07:38.257428 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.257468 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.257484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.257522 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.258241 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44896: BlockingRecv error: recv got EOF from 127.0.0.1:44896 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 10:07:38.259198 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.256322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.19.46.253:42891)
0925 10:07:38.256364 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.256367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.258621 (+ 2254us) server_negotiation.cc:407] Connection header received
0925 10:07:38.258725 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.258727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.258739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.258768 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.259126 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46270: BlockingRecv error: recv got EOF from 127.0.0.1:46270 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:07:38.260053 10038 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.257007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.19.46.252:45281)
0925 10:07:38.257052 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.257055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.259469 (+ 2414us) server_negotiation.cc:407] Connection header received
0925 10:07:38.259565 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.259567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.259578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.259607 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.259978 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48510: BlockingRecv error: recv got EOF from 127.0.0.1:48510 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 10:07:38.262956 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.261796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.19.46.253:42891)
0925 10:07:38.261838 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.261840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.262257 (+ 417us) server_negotiation.cc:407] Connection header received
0925 10:07:38.262418 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.262420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.262444 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.262475 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.262902 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46280: BlockingRecv error: recv got EOF from 127.0.0.1:46280 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:07:38.266546 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.265600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.19.46.253:42891)
0925 10:07:38.265641 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.265643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.265937 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:07:38.266061 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.266063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.266075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.266105 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.266473 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46286: BlockingRecv error: recv got EOF from 127.0.0.1:46286 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:07:38.274935 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.273914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.19.46.253:42891)
0925 10:07:38.273963 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.273966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.274301 (+ 335us) server_negotiation.cc:407] Connection header received
0925 10:07:38.274414 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.274415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.274427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.274459 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.274869 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46292: BlockingRecv error: recv got EOF from 127.0.0.1:46292 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:07:38.281493 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.280401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.19.46.253:42891)
0925 10:07:38.280447 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.280449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.280743 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:07:38.280864 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.280865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.280877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.280908 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.281405 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46304: BlockingRecv error: recv got EOF from 127.0.0.1:46304 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:07:38.291857 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.290901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.19.46.253:42891)
0925 10:07:38.290946 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.290948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.291220 (+ 272us) server_negotiation.cc:407] Connection header received
0925 10:07:38.291328 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.291330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.291341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.291371 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.291747 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46318: BlockingRecv error: recv got EOF from 127.0.0.1:46318 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:07:38.310315 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.309222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.19.46.253:42891)
0925 10:07:38.309316 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.309318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.309650 (+ 332us) server_negotiation.cc:407] Connection header received
0925 10:07:38.309758 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.309759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.309771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.309823 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.310250 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46332: BlockingRecv error: recv got EOF from 127.0.0.1:46332 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 10:07:38.344858 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.343776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.19.46.253:42891)
0925 10:07:38.343835 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.343837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.344135 (+ 298us) server_negotiation.cc:407] Connection header received
0925 10:07:38.344273 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.344274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.344288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.344328 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.344766 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46340: BlockingRecv error: recv got EOF from 127.0.0.1:46340 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:07:38.411180 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.410376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.19.46.253:42891)
0925 10:07:38.410444 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.410446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.410563 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:07:38.410640 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.410641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.410652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.410696 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.411130 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46350: BlockingRecv error: recv got EOF from 127.0.0.1:46350 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 10:07:38.541498 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.540629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.19.46.253:42891)
0925 10:07:38.540724 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.540727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.540798 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:07:38.540910 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.540911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.540923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.541011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.541372 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46356: BlockingRecv error: recv got EOF from 127.0.0.1:46356 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:07:38.799659 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:07:38.798892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.19.46.253:42891)
0925 10:07:38.798994 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:38.798996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:38.799007 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:07:38.799073 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:38.799073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:38.799083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:38.799135 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:38.799579 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46368: BlockingRecv error: recv got EOF from 127.0.0.1:46368 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:07:39.172451 10066 catalog_manager.cc:1259] Loaded cluster ID: 224617c3aff14d53be2a38914b7625d3
I20260925 10:07:39.172529 10066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: loading cluster ID for follower catalog manager: success
I20260925 10:07:39.173300 10066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: acquiring CA information for follower catalog manager: success
I20260925 10:07:39.173643 10066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1d526f4570c40af9939cfc1b7f6c645: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:07:39.218739 10095 catalog_manager.cc:1259] Loaded cluster ID: 224617c3aff14d53be2a38914b7625d3
I20260925 10:07:39.218818 10095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: loading cluster ID for follower catalog manager: success
I20260925 10:07:39.219662 10095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: acquiring CA information for follower catalog manager: success
I20260925 10:07:39.219954 10095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a456ab21a3a47f5a102aef804dbda95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:07:39.315878 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:39.312886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.19.46.253:42891)
0925 10:07:39.313028 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:39.313030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:39.313082 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:07:39.313152 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:39.313152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:39.313202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:39.313268 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:39.313481 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.314007 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.314897 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.315099 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.315163 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:39.315408 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:39.315602 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:39.315701 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:39.315737 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 10:07:39.315759 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260925 10:07:39.320685 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:39.318138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.19.46.252:45281)
0925 10:07:39.318239 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:39.318241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:39.318257 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:07:39.318293 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:39.318294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:39.318333 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:39.318418 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:39.318534 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.319013 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.319797 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.320024 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.320081 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:39.320210 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:39.320317 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:39.320373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:39.320595 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:07:39.320616 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260925 10:07:39.325736 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:39.322815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.19.46.252:45281)
0925 10:07:39.322880 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:39.322882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:39.323040 (+ 158us) server_negotiation.cc:407] Connection header received
0925 10:07:39.323113 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:39.323114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:39.323124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:39.323173 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:39.323316 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.324000 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.324860 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.325138 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.325203 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:39.325354 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:39.325386 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:39.325453 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:39.325613 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 10:07:39.325635 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:07:39.325904 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:39.322545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.19.46.253:42891)
0925 10:07:39.322589 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:39.322591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:39.322693 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:07:39.322791 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:39.322792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:39.322804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:39.322862 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:39.323030 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.323818 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.324934 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.325234 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.325297 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:39.325494 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:39.325523 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:39.325584 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:39.325734 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:07:39.325762 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:07:39.326437 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:39.322486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.19.46.254:37685)
0925 10:07:39.322590 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:39.322591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:39.322658 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:07:39.322754 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:39.322755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:39.322811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:39.322909 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:07:39.323137 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.323746 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.325354 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:39.325667 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:39.325726 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:39.326046 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:07:39.326162 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:07:39.326205 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:39.326323 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:07:39.326350 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
I20260925 10:07:39.328444 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9813
I20260925 10:07:39.334409 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9889
I20260925 10:07:39.339716 19643 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task46RfQ8/build/release/bin/kudu with pid 9965
2026-09-25T10:07:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2881 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5850 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (303063 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST